Sparse Regularized Optimal Transport with Deformed q-Entropy

Abstract

Optimal transport is a mathematical tool that has been a widely used to measure the distance between two probability distributions. To mitigate the cubic computational complexity of the vanilla formulation of the optimal transport problem, regularized optimal transport has received attention in recent years, which is a convex program to minimize the linear transport cost with an added convex regularizer. Sinkhorn optimal transport is the most prominent one regularized with negative Shannon entropy, leading to densely supported solutions, which are often undesirable in light of the interpretability of transport plans. In this paper, we report that a deformed entropy designed by q-algebra, a popular generalization of the standard algebra studied in Tsallis statistical mechanics, makes optimal transport solutions supported sparsely. This entropy with a deformation parameter q interpolates the negative Shannon entropy ($q=1$) and the squared 2-norm ($q=0$), and the solution becomes more sparse as q tends to zero. Our theoretical analysis reveals that a larger q leads to a faster convergence when optimized with the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm. In summary, the deformation induces a trade-off between the sparsity and convergence speed.

1. Introduction

Optimal transport (OT) is a classic problem in operations research, and it is used to compute a transport plan between suppliers and demanders with a minimum transportation cost. The minimum transportation cost can be interpreted as the closeness between the distributions when considering suppliers and demanders as two probability distributions. The OT problem has been extensively studied (also as the Wasserstein distance) [1] and used in robust machine learning [2], domain adaptation [3], generative modeling [4], and natural language processing [5], attributed to its many useful properties, such as the distance between two probability distributions. Recently, the OT problem has been employed for various modern applications, such as interpretable word alignment [6] and the locality-aware evaluation of object detection [7], because it can capture the geometry of data and provide a measurement method for closeness and alignment among different objects. From a computational perspective, a naïve approach is to use a network simplex algorithm or interior point method to solve the OT problem as a usual linear program; this approach requires supercubic time complexity [8] and is not scalable. A number of approaches have been suggested to accelerate the computation of the OT problem: entropic regularization [9,10], accelerated gradient descent [11], and approximation with tree [12] and graph metrics [13]. We focused our attention on entropic-regularized OT because it allows a unique solution attributed to strong convexity and transforms the original constrained optimization into an unconstrained problem with a clear primal–dual relationship. The celebrated Sinkhorn algorithm solves entropic-regularized OT with square-time complexity [9]. Furthermore, the Sinkhorn algorithm is amenable to differentiable programming, and it is easily incorporated into end-to-end learning pipelines [14,15].

Despite the popularity of the Sinkhorn algorithm, one of the main drawback is that Shannon entropy blurs the OT solution, i.e., solutions of entropic-regularized OT are always densely supported. The Shannon entropy induces a probability distribution that has strictly positive values everywhere on its support owing to the nature of the Shannon entropy [16] whereas the vanilla (unregularized) OT produces extremely sparse transport plans located on the boundaries of a polytope [17,18]. If we are interested in alignment and matching between different objects (such as in the several applications of natural language processing [6,19]), dense transport plans are not so interpretable that matching information between objects may be obfuscated by unimportant small densities contained in the transport plans. One attempt toward realizing sparse OT is to use the squared two-norm as an alternative regularizer. Blondel et al. [20] showed that the dual of this optimization problem can be solved via the L-BFGS method [21]; the primal solution corresponds to a transport plan recovered from the dual solution in a closed form, which is sparse. Although they successfully obtained a sparse OT formulation with a numerically stable algorithm, the degree of the sparsity cannot be easily modulated when we prefer to control the sparsity given a final application. Furthermore, the theoretical convergence rates of solving regularized OT are yet to be known.

In this study, we aimed to examine the relationship between the sparsity of transport plans and the convergence guarantee of regularized OT. Specifically, we propose yet another entropic regularizer called deformed q-entropy with a deformation parameter q that allows us to control the solution sparsity. We start with a dual solution of the entropic-regularized OT given by the Gibbs kernel to introduce a new regularizer; the Gibbs kernel associated with Shannon entropy induces nonsparsity, and, therefore, we replace the Gibbs kernel with another sparse kernel based on q-exponential distribution [22], following the idea of Tsallis statistics [23]. The deformed q entropy is derived from the dual solution characterized by the sparse kernel. Interestingly, the deformed q entropy recovers the Shannon entropy at the limit of $q\nearrow 1$ and matches the (negative) squared two-norm at $q=0$; this means that the deformed q entropy interpolates between the two regularizers. We confirm that the solution becomes increasingly sparse as q approaches zero. We call the regularized OT with the deformed q entropy deformed q-optimal transport (q-DOT). The q-DOT reveals an interesting connection between the OT solution and the q-exponential distribution, which is an independent interest. From the optimization perspective, we can solve the unconstrained dual of q-DOT with many standard solvers, as reported in Blondel et al. [20]. We can see that the convergence becomes faster with the BFGS method [24] as the deformation parameter q approaches one, as a result of our analysis of the convergence rate of the dual optimization. Therefore, the weaker deformation (larger q) leads to faster convergence while sacrificing sparsity. Finally, we demonstrate the trade-off between sparsity and convergence in the numerical experiments.

Our contributions can be summarized as: (i) showing a clear connection between the regularized OT problem and the q-exponential distribution; (ii) demonstrating the trade-off of the q-DOT between sparsity and convergence; (iii) providing a formal convergence guarantee of the q-DOT when solved with the BFGS method. The rest of this paper is organized as follows: Section 2 introduces the necessary background to the OT problem and entropic regularization. In Section 3, the Lagrange dual of the entropic-regularized OT problem is first shown; then, the dual optimal formula and the q-exponential distribution is connected to sparsify the transport matrix. Section 4 specifically focuses on the optimization perspective of the regularized OT problem, and a convergence guarantee with the BFGS method is provided, which shows the theoretical trade-off between sparsity and convergence. Finally, the empirical behavior and the trade-off of the regularized OT are numerically confirmed in Section 5.

2. Background

2.1. Preliminaries

For $x\in \mathbb{R}$, let ${\left[x\right]}_{+}=x$ if $x>0$ and 0 otherwise, and let ${\left[x\right]}_{+}^p$ represent ${\left({\left[x\right]}_{+}\right)}^p$ hereafter. For a convex function $f,\mathcal{X}\to \mathbb{R}$, where $\mathcal{X}$ represents a Euclidean vector space equipped with an inner product $〈·,·〉$, the Fenchel–Legendre conjugate $f^{\star }:\mathcal{X}\to \mathbb{R}$ is defined as $f^{\star }\left(\mathbf{y}\right){sup}_{\mathbf{x}\in \mathcal{X}}〈\mathbf{x},\mathbf{y}〉-f\left(\mathbf{x}\right)$. The relative interior of a set S is denoted by $riS$, and the effective domain of a function f is denoted by $dom\left(f\right)$. A differentiable function f is said to be M-strongly convex over $S\subseteq ridom\left(f\right)$ if, for all $\mathbf{x},\mathbf{y}\in S$, we have $f\left(\mathbf{x}\right)-f\left(\mathbf{y}\right)\le 〈\nabla f\left(\mathbf{x}\right),\mathbf{x}-\mathbf{y}〉-\frac{M}{2}{\parallel \mathbf{x}-\mathbf{y}\parallel }_2^2$. If f is twice differentiable, the strong convexity is equivalent to ${\nabla }^{2}f\left(\mathbf{x}\right)⪰M\mathbf{I}$ for all $\mathbf{x}\in S$. Similarly, a differentiable function f is said to be M-smooth over $S\subseteq ridom\left(f\right)$ if for all $\mathbf{x},\mathbf{y}\in S$, we have ${\parallel \nabla f\left(\mathbf{x}\right)-\nabla f\left(\mathbf{y}\right)\parallel }_2\le M{\parallel \mathbf{x}-\mathbf{y}\parallel }_2$, which is equivalent to ${\nabla }^{2}f\left(\mathbf{x}\right)⪯M\mathbf{I}$ for all $\mathbf{x}\in S$ if f is twice differentiable.

2.2. Optimal Transport

The OT is a mathematical problem to find a transport plan between two probability distributions with the minimum transport cost. The discussions in this paper are restricted to discrete distributions. Let $(\mathcal{X},d)$, ${\delta }_{\mathbf{x}}$, and ${▵}^{n-1}:=\left\{\mathbf{p}\in {[0,1]}^n|〈\mathbf{p},{\mathbf{1}}_n〉=1\right\}$ represent a metric space, Dirac measure at point $\mathbf{x}$, and $(n-1)$-dimensional probability simplex, respectively. Let $\mu ={\sum }_{i=1}^n{\mathbf{a}}_i{\delta }_{{\mathbf{x}}_i}$ and $\nu ={\sum }_{j=1}^m{\mathbf{b}}_i{\delta }_{{\mathbf{y}}_i}$ be histograms supported on the finite sets of points ${\left({\mathbf{x}}_i\right)}_{i=1}^n\subseteq \mathcal{X}$ and ${\left({\mathbf{y}}_j\right)}_{j=1}^m\subseteq \mathcal{X}$, respectively, where $\mathbf{a}\in {▵}^{n-1}$ and $\mathbf{b}\in {▵}^{m-1}$ are probability vectors. The OT between two discrete probability measures $\mu$ and $\nu$ is the optimization problem

$$\mathcal{T}(\mu ,\nu ):=\underset{\Pi \in \mathcal{U}(\mu ,\nu )}{inf}\sum _{i=1}^n\sum _{j=1}^{m}d({\mathbf{x}}_i,{\mathbf{y}}_j){\mathbf{\Pi }}_{ij},$$

(1)

where $\mathcal{U}$ represents the transport polytope, defined as

$$\mathcal{U}(\mu ,\nu ):=\left\{\mathbf{\Pi }\in {\mathbb{R}}_{\ge 0}^{n\times m}|\mathbf{\Pi }{\mathbf{1}}_m=\mathbf{a},{\mathbf{\Pi }}^{\top }{\mathbf{1}}_n=\mathbf{b}\right\}.$$

(2)

The transport polytope $\mathcal{U}$ defines the constraints on the row/column marginals of a transport matrix $\mathbf{\Pi }$. These constraints are often referred to as coupling constraints. For notational simplicity, matrix ${\mathbf{D}}_{ij}:=d({\mathbf{x}}_i,{\mathbf{y}}_j)$ and expectation $〈\mathbf{D},\mathbf{\Pi }〉:={\sum }_{i=1}^n{\sum }_{j=1}^m{\mathbf{D}}_{ij}{\mathbf{\Pi }}_{ij}$ are used hereafter. $\mathcal{T}(\mu ,\nu )$ is known as a 1-Wasserstein distance, which defines a metric space over histograms [1].

Equation (1) is a linear program and can be solved by well-studied algorithms such as the interior point and network simplex methods. However, its computational complexity is $\mathcal{O}(n^{3}logn)$ (assuming $n=m$), so is not scalable to large datasets [8].

2.3. Entropic Regularization and Sinkhorn Algorithm

The entropic-regularized formulation is commonly used to reduce the computational burden. Here, we introduce regularized OT with negative Shannon entropy [9] as

$${\mathcal{T}}_{-\lambda H}(\mu ,\nu ):=\underset{\mathbf{\Pi }\in \mathcal{U}(\mu ,\nu )}{inf}〈\mathbf{D},\mathbf{\Pi }〉+\lambda \underset{\mathrm{negative}\mathrm{Shannon}\mathrm{entropy}}{\underbrace{\sum _{i=1}^n\sum _{j=1}^m({\mathbf{\Pi }}_{ij}log{\mathbf{\Pi }}_{ij}-{\mathbf{\Pi }}_{ij})}},$$

(3)

where $\lambda >0$ represents the regularization strength. Let us review the derivation of the updates of the Sinkhorn algorithm. The Lagrangian of the optimization problem in Equation (3) is

$$\begin{array}{cc}\hfill \mathcal{L}(\mathbf{\Pi },\mathbf{\alpha },\mathbf{\beta }):=& \sum _{i=1}^n\sum _{j=1}^m({\mathbf{D}}_{ij}{\mathbf{\Pi }}_{ij}+\lambda ({\mathbf{\Pi }}_{ij}log{\mathbf{\Pi }}_{ij}-{\mathbf{\Pi }}_{ij}))\hfill \\ \hfill & +\sum _{i=1}^n{\mathbf{\alpha }}_i({[\mathbf{\Pi }{\mathbf{1}}_m]}_i-{\mathbf{a}}_i)+\sum _{j=1}^m{\mathbf{\beta }}_j({\left[{\mathbf{\Pi }}^{\top }{\mathbf{1}}_n\right]}_j-{\mathbf{b}}_j),\hfill \end{array}$$

(4)

where $\mathbf{\alpha }\in {\mathbb{R}}^n$ and $\mathbf{\beta }\in {\mathbb{R}}^m$ represent the Lagrangian multipliers. Equation (4) ignores the constraints ${\mathbf{\Pi }}_{ij}\ge 0$ (for all $i\in \left[n\right]$ and $j\in \left[m\right]$); however, they will be automatically satisfied. By taking the derivative in ${\mathbf{\Pi }}_{ij}$,

$${\nabla }_{{\mathbf{\Pi }}_{ij}}\mathcal{L}={\mathbf{D}}_{ij}+\lambda log{\mathbf{\Pi }}_{ij}+{\mathbf{\alpha }}_i+{\mathbf{\beta }}_j,$$

(5)

and, hence, the stationary condition ${\nabla }_{{\mathbf{\Pi }}_{ij}}\mathcal{L}=0$ induces the solution

$${\mathbf{\Pi }}_{ij}=exp\left(-\frac{{\mathbf{\alpha }}_i+{\mathbf{\beta }}_j+{\mathbf{D}}_{ij}}{\lambda }\right).$$

(6)

The decomposition ${\mathbf{\Pi }}_{ij}=exp\left(-\frac{{\mathbf{D}}_{ij}}{\lambda }\right)/exp\left(\frac{{\mathbf{\alpha }}_i+{\mathbf{\beta }}_j}{\lambda }\right)$ suggests that the stationary point is the (normalized) Gibbs kernel $exp\left(-\frac{{\mathbf{D}}_{ij}}{\lambda }\right)$. One can easily infer that the Sinkhorn solution is dense because the Gibbs kernel is supported on the entire ${\mathbb{R}}_{\ge 0}$, i.e., $exp\left(-\frac{z}{\lambda }\right)>0$ for all $z\in {\mathbb{R}}_{\ge 0}$. We can write Equation (6) into a matrix form by applying the variable transforms ${\mathbf{u}}_i:=exp\left(-\frac{{\mathbf{\alpha }}_i}{\lambda }\right)$, ${\mathbf{v}}_j:=exp\left(-\frac{{\mathbf{\beta }}_j}{\lambda }\right)$, and ${\mathbf{K}}_{ij}:=exp\left(-\frac{{\mathbf{D}}_{ij}}{\lambda }\right)$ as

$$\mathbf{\Pi }=\underset{:=\mathbf{U}}{\underbrace{diag\left(\mathbf{u}\right)}} \mathbf{K}\underset{:=\mathbf{V}}{\underbrace{diag\left(\mathbf{v}\right)}}.$$

(7)

The following Sinkhorn updates are used to make Equation (7) meet the marginal constraints:

$$\left\{\begin{array}{c}{\mathbf{u}}^{\prime }\leftarrow \mathbf{a}/\left(\mathbf{K}\mathbf{v}\right)\hfill \\ {\mathbf{v}}^{\prime }\leftarrow \mathbf{b}/\left({\mathbf{K}}^{\top }\mathbf{u}\right)\hfill \end{array}\right.,$$

(8)

where $\mathbf{z}/\mathbf{\eta }$ represents the element-wise division of the two vectors $\mathbf{z}$ and $\mathbf{\eta }$. The computational complexity is $\mathcal{O}\left(Knm\right)$ because the Sinkhorn updates involve only matrix-vector multiplications and element-wise divisions; K represents the number of the Sinkhorn updates. Finer analysis of the number of updates required to meet the error tolerance is provided in the literature [25].

3. Deformed q-Entropy and q-Regularized Optimal Transport

3.1. Regularized Optimal Transport and Its Dual

Let us consider the following primal problem with a general regularization function $\Omega$.

Definition 1

(Primal of regularized OT).

$${\mathcal{T}}_{\Omega }(\mu ,\nu )=\underset{\mathbf{\Pi }\in \mathcal{U}(\mu ,\nu )}{inf}〈\mathbf{D},\mathbf{\Pi }〉+\sum _{i,j}\Omega \left({\mathbf{\Pi }}_{ij}\right),$$

(9)

where $\Omega :\mathbb{R}\to \mathbb{R}$ represents a proper closed convex function.

Next, we derive its dual by Lagrange duality. The Lagrangian of Equation (9) is defined as

$$\mathcal{L}(\mathbf{\Pi },\mathbf{\alpha },\mathbf{\beta }):=〈\mathbf{D},\mathbf{\Pi }〉+\sum _{i,j}\Omega \left({\mathbf{\Pi }}_{ij}\right)+〈\mathbf{\alpha },\mathbf{\Pi }{\mathbf{1}}_m-\mathbf{a}〉+〈\mathbf{\beta },{\mathbf{\Pi }}^{\top }{\mathbf{1}}_n-\mathbf{b}〉,$$

(10)

with dual variables $\mathbf{\alpha }\in {\mathbb{R}}^n$ and $\mathbf{\beta }\in {\mathbb{R}}^m$. Then, the primal can be rewritten in terms of the Lagrangian

$${\mathcal{T}}_{\Omega }(\mu ,\nu )=\underset{\mathbf{\Pi }\in {\mathbb{R}}_{\ge 0}^{n\times m}}{inf}\underset{\mathbf{\alpha }\in {\mathbb{R}}^n,\mathbf{\beta }\in {\mathbb{R}}^m}{sup}\mathcal{L}(\mathbf{\Pi },\mathbf{\alpha },\mathbf{\beta }).$$

(11)

In this Lagrangian formulation, we let the constraints $\mathbf{\Pi }\in {\mathbb{R}}_{\ge 0}^{n\times m}$ remain for a technical reason. The constrained optimization problem in (11) can be reformulated into the following unconstrained one with an indicator function $I_{{\mathbb{R}}_{\ge 0}^{n\times m}}$.

$${\mathcal{T}}_{\Omega }(\mu ,\nu )=\underset{\mathbf{\Pi }\in {\mathbb{R}}^{m\times m}}{inf}\underset{\mathbf{\alpha }\in {\mathbb{R}}^n,\mathbf{\beta }\in {\mathbb{R}}^m}{sup}\mathcal{L}(\mathbf{\Pi },\mathbf{\alpha },\mathbf{\beta })+I_{{\mathbb{R}}_{\ge 0}^{n\times m}}(\mathbf{\Pi }),$$

(12)

which corresponds to an optimization problem with the convex objective function $〈\mathbf{D},\mathbf{\Pi }〉+{\sum }_{i,j}\Omega \left({\mathbf{\Pi }}_{ij}\right)+I_{{\mathbb{R}}_{\ge 0}^{n\times m}}(\mathbf{\Pi })$ with only the linear constraints $\mathbf{\Pi }{\mathbf{1}}_m=\mathbf{a}$ and ${\mathbf{\Pi }}^{\top }{\mathbf{1}}_n=\mathbf{b}$. By invoking the Sinkhorn–Knopp theorem [26], the existence of a strictly feasible solution, namely, a solution satisfying $\mathbf{\Pi }{\mathbf{1}}_m=\mathbf{a}$ and ${\mathbf{\Pi }}^{\top }{\mathbf{1}}_n=\mathbf{b}$, can be confirmed. Hence, we see that the Slater condition is satisfied, and the strong duality holds as follows:

$$\begin{array}{cc}\hfill {\mathcal{T}}_{\Omega }& (\mu ,\nu )=\underset{\mathbf{\alpha }\in {\mathbb{R}}^n,\mathbf{\beta }\in {\mathbb{R}}^m}{sup}\underset{\mathbf{\Pi }\in {\mathbb{R}}_{\ge 0}^{n\times m}}{inf}\mathcal{L}(\mathbf{\Pi },\mathbf{\alpha },\mathbf{\beta })\hfill \\ \hfill & =\underset{\mathbf{\alpha }\in {\mathbb{R}}^n,\mathbf{\beta }\in {\mathbb{R}}^m}{sup}-〈\mathbf{a},\mathbf{\alpha }〉-〈\mathbf{b},\mathbf{\beta }〉+\underset{\mathbf{\Pi }\in {\mathbb{R}}_{\ge 0}^{n\times m}}{inf}\sum _{i,j}({\mathbf{D}}_{ij}+{\mathbf{\alpha }}_i+{\mathbf{\beta }}_j){\mathbf{\Pi }}_{ij}+\Omega \left({\mathbf{\Pi }}_{ij}\right)\hfill \\ \hfill & =\underset{\mathbf{\alpha }\in {\mathbb{R}}^n,\mathbf{\beta }\in {\mathbb{R}}^m}{sup}-〈\mathbf{a},\mathbf{\alpha }〉-〈\mathbf{b},\mathbf{\beta }〉-\left(\underset{\mathbf{\Pi }\in {\mathbb{R}}_{\ge 0}^{n\times m}}{sup}\sum _{i,j}-({\mathbf{D}}_{ij}+{\mathbf{\alpha }}_i+{\mathbf{\beta }}_j){\mathbf{\Pi }}_{ij}-\Omega \left({\mathbf{\Pi }}_{ij}\right)\right)\hfill \\ \hfill & =\underset{\mathbf{\alpha }\in {\mathbb{R}}^n,\mathbf{\beta }\in {\mathbb{R}}^m}{sup}-〈\mathbf{a},\mathbf{\alpha }〉-〈\mathbf{b},\mathbf{\beta }〉-\sum _{i,j}{\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j),\hfill \end{array}$$

(13)

where ${\Omega }^{\star }$ represents the Fenchel–Legendre conjugate of $\Omega :\mathbb{R}\to \mathbb{R}$

$${\Omega }^{\star }\left(\eta \right):=\underset{\pi \ge 0}{sup}\eta \pi -\Omega \left(\pi \right).$$

(14)

Although each element of the transport plans ranges over $[0,1]$, it is sufficient to define the Fenchel–Legendre conjugate as the supremum over ${\mathbb{R}}_{\ge 0}$ because of how ${\Omega }^{\star }$ emerges in the strong duality (13). According to Danskin’s theorem [27], the supremum of the Fenchel–Legendre conjugate can be attained at

$${\mathbf{\Pi }}_{ij}^{\star }=\nabla {\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j).$$

(15)

Therefore, the dual of regularized OT is formulated as follows:

Definition  2

(Dual of regularized OT).

$${\mathcal{T}}_{\Omega }(\mu ,\nu )=\underset{\mathbf{\alpha }\in {\mathbb{R}}^n,\mathbf{\beta }\in {\mathbb{R}}^m}{sup}-〈\mathbf{a},\mathbf{\alpha }〉-〈\mathbf{b},\mathbf{\beta }〉-\sum _{i,j}{\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j),$$

(16)

where ${\Omega }^{\star }$ represents the Fenchel–Legendre conjugate ${\Omega }^{\star }\left(\eta \right):={sup}_{\pi \ge 0}\eta \pi -\Omega \left(\pi \right)$. The optimal solution of the primal is given by the dual map $\nabla {\Omega }^{\star }$ such that ${\mathbf{\Pi }}_{ij}^{\star }=\nabla {\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i^{\star }-{\mathbf{\beta }}_j^{\star })$, where $({\mathbf{\alpha }}^{\star },{\mathbf{\beta }}^{\star })$ represents the dual optimal solution.

Next, we see several examples that are summarized in

Table 1. Summary of $\Omega \left(\pi \right)$, ${\Omega }^{\star }\left(\eta \right)$, and $\nabla {\Omega }^{\star }\left(\eta \right)$ for several regularizers. The relationship between $\Omega$, its conjugate, and the derivatives are summarized in Bao and Sugiyama [28].

	$\mathbf{\Omega }\left(\mathit{\pi }\right)$	${\mathbf{\Omega }}^{\star }\left(\mathit{\eta }\right)$	$\mathbf{\nabla }{\mathbf{\Omega }}^{\star }\left(\mathit{\eta }\right)$
Negative entropy	$\lambda (\pi log\pi -\pi )$	$\lambda e^{\eta /\lambda }$	$e^{\eta /\lambda }$
Squared 2-norm	$\frac{\lambda }{2}{\pi }^2$	$\frac{1}{2\lambda }{\left[\eta \right]}_{+}^2$	$\frac{1}{\lambda }{\left[\eta \right]}_{+}$
Deformed q entropy	$\frac{\lambda }{2-q}(\pi {log}_q\left(\pi \right)-\pi )$	$\frac{\lambda }{2-q}{exp}_q{(\eta /\lambda )}^{2-q}$	${exp}_q(\eta /\lambda )$

.

Example 1

(Negative Shannon entropy). Let $\Omega \left(\pi \right)=-\lambda H\left(\pi \right)=\lambda (\pi log\pi -\pi )$; then ${\Omega }^{\star }\left(\eta \right)=\lambda e^{\eta /\lambda }$ and $\nabla {\Omega }^{\star }\left(\eta \right)=e^{\eta /\lambda }$. The optimal solution represented with the optimal dual variables $({\mathbf{\alpha }}^{\star },{\mathbf{\beta }}^{\star })$ is ${\mathbf{\Pi }}_{ij}^{\star }=exp\left(-\frac{{\mathbf{D}}_{ij}+{\mathbf{\alpha }}_i^{\star }+{\mathbf{\beta }}_j^{\star }}{\lambda }\right)$. This recovers the stationary point of the Sinkhorn OT in Equation (6). The solution is dense because the regularizer Ω induces the Gibbs kernel $\nabla {\Omega }^{\star }\left(\eta \right)=e^{\eta /\lambda }>0$ for all $\eta \in \mathbb{R}$.

Example 2

(Squared 2-norm). Let $\Omega \left(\pi \right)=\frac{\lambda }{2}{\pi }^2$; then ${\Omega }^{\star }\left(\eta \right)=\frac{1}{2\lambda }{\left[\eta \right]}_{+}^2$ and $\nabla {\Omega }^{\star }\left(\eta \right)=\frac{1}{\lambda }{\left[\eta \right]}_{+}$. The optimal solution represented with the optimal dual variables $({\mathbf{\alpha }}^{\star },{\mathbf{\beta }}^{\star })$ is ${\mathbf{\Pi }}_{ij}^{\star }=\frac{1}{\lambda }{\left[-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i^{\star }-{\mathbf{\beta }}_j^{\star }\right]}_{+}$. As mentioned by Blondel et al. [20], the squared 2-norm can sparsify the solution because $\nabla {\Omega }^{\star }\left(\eta \right)=\frac{1}{\lambda }{\left[\eta \right]}_{+}$ may take the value 0.

3.2. q Algebra and Deformed Entropy

As shown in the last few examples, the dual map $\nabla {\Omega }^{\star }$ plays an important role in the OT solution sparsity. In addition, the induced $\nabla {\Omega }^{\star }$ is the Gibbs kernel when the negative Shannon entropy is used as $\Omega$. Therefore, one may think of designing a regularizer from $\nabla {\Omega }^{\star }$ by utilizing a kernel function that induces sparsity. One candidate is a q-exponential distribution. We begin with some basics required to formulate q-exponential distributions.

First, we introduce q-algebra, which has been well studied in the field of Tsallis statistical mechanics [23,29,30]. q algebra has been used in the machine-learning literature for regression [31], Bayesian inference [32], and robust learning [33]. For a deformation parameter $q\in [0,1]$, the q-logarithm and q-exponential functions are defined as

$$\begin{array}{ccc}\hfill {log}_q\left(x\right):=\left\{\begin{array}{cc}\frac{x^{1-q}-1}{1-q}\hfill & \mathrm{if}q\in [0,1)\hfill \\ log\left(x\right)\hfill & \mathrm{if}q=1\hfill \end{array}\right.,& \hfill & \hfill {exp}_q\left(x\right):=\left\{\begin{array}{cc}{[1+(1-q)x]}_{+}^{1/(1-q)}\hfill & \mathrm{if}q\in [0,1)\hfill \\ exp\left(x\right)\hfill & \mathrm{if}q=1\hfill \end{array}\right..\end{array}$$

(17)

The q logarithm is defined for only $x>0$, as in the natural logarithm; they are inverse functions to each other (in an appropriate domain) and they recover the natural definition of the logarithm and exponential as $q\nearrow 1$. Their derivatives are ${\left({log}_q\left(x\right)\right)}^{\prime }=\frac{1}{x^q}$ and ${\left({exp}_q\left(x\right)\right)}^{\prime }={exp}_q{\left(x\right)}^q$, respectively. The additive factorization property $exp(x+y)=exp\left(x\right)exp\left(y\right)$ satisfied by the natural exponential no longer holds for the q exponential, such that ${exp}_q(x+y)\ne {exp}_q\left(x\right){exp}_q\left(y\right)={exp}_q(x+y+(1-q)xy)$. Instead, we can construct another algebraic structure by introducing the other operation called the q product ${\otimes }_q$:

$$x{\otimes }_{q}y={[x^{1-q}+y^{1-q}-1]}_{+}^{1/(1-q)}.$$

(18)

With this product, the pseudoadditive factorization ${exp}_q(x+y)={exp}_q\left(x\right){\otimes }_q{exp}_q\left(y\right)$ holds. Thus, the q algebra captures rich nonlinear structures, and it is often used to extend the Shannon entropy to the Tsallis entropy [23]

$$T_q\left(\mathbf{\pi }\right)=-\sum _{i=1}^n{\mathbf{\pi }}_i^q{log}_q\left({\mathbf{\pi }}_i\right).$$

(19)

One can see that the Tsallis entropy has an equivalent power formulation $T_q\left(\mathbf{\pi }\right)={\sum }_{i=1}^n\frac{{\mathbf{\pi }}_i-{\mathbf{\pi }}_i^q}{1-q}$, which means that it is often suitable for modeling heavy-tailed phenomena such as the power law. Although the introduced q logarithm and exponential can look arbitrary, they can be axiomatically derived by assuming the essential properties of the algebra (see Naudts [29]). For more physical insights, we recommend readers to refer to the literature [30].

Next, we introduce the q-exponential distribution. We introduce a simpler form for our purpose, whereas more general formulations of the q-exponential distribution have been introduced in the literature [22]. Given the form of the Gibbs kernel $k\left(\xi \right):=exp(-\xi /\lambda )$, we define the q-Gibbs kernel as follows:

Definition  3

(q-Gibbs kernel). For $\xi \ge 0$, we define the q-Gibbs kernel as $k_q\left(\xi \right){exp}_q(-\xi /\lambda )$ for a deformation parameter $q\in [0,1]$ and a temperature parameter $\lambda \in {\mathbb{R}}_{>0}$.

If we take $\xi$ as the (centered) squared distance, then $k_q\left(\xi \right)$ represents the q-Gaussian distribution [22]. We illustrate the q-Gibbs kernel with different deformation parameters in Figure 1.

By definition, the support of the q-Gibbs kernel is $supp\left(k_q\right)=\left[0,\frac{\lambda }{1-q}\right]$ for $q\in [0,1)$ and $supp\left(k_q\right)={\mathbb{R}}_{\ge 0}$ for $q=1$. This indicates that the q-Gibbs kernel ignores the effect of a too-large $\xi$ (or too large a distance between two points); its threshold is smoothly controlled by the temperature parameter $\lambda$ and deformation parameter q.

Finally, we derive an entropic regularizer that induces sparsity by using the q-Gibbs kernel. Given the stationary condition in Equation (15), we impose the following functional form on the dual map:

$$\pi =\nabla {\Omega }^{\star }\left(\eta \right)={exp}_q\left(\frac{\eta }{\lambda }\right),$$

(20)

where $(\pi ,\eta )=({\mathbf{\Pi }}_{ij}^{\star },-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j)$. Equation (20) results in the factorization

$${\mathbf{\Pi }}_{ij}^{\star }={exp}_q\left(-\frac{{\mathbf{\alpha }}_i}{\lambda }\right){\otimes }_q{exp}_q\left(-\frac{-{\mathbf{D}}_{ij}}{\lambda }\right){\otimes }_q{exp}_q\left(-\frac{-{\mathbf{\beta }}_j}{\lambda }\right),$$

(21)

and a sufficiently large input distance ${\mathbf{D}}_{ij}$ drives ${\mathbf{\Pi }}_{ij}$ to zero; though ${exp}_q(-{\mathbf{D}}_{ij}/\lambda )=0$ does not immediately imply ${\mathbf{\Pi }}_{ij}^{\star }=0$ because the q-product ${\otimes }_q$ lacks an absorbing element. By solving Equation (20),

$$\nabla \Omega \left(\pi \right)=\lambda {log}_q\left(\pi \right),\Omega \left(\pi \right)=\frac{\lambda }{2-q}\left(\pi {log}_q\left(\pi \right)-\pi \right).$$

(22)

For the completeness, its derivation is shown in Appendix A. Hence, we define the deformed q entropy as follows:

Definition  4

(Deformed q-entropy). For $\mathbf{\pi }\in {▵}^{n-1}$, the deformed q entropy is defined as

$$H_q\left(\mathbf{\pi }\right)=-\frac{1}{2-q}\sum _{i=1}^n({\mathbf{\pi }}_i{log}_q\left({\mathbf{\pi }}_i\right)-{\mathbf{\pi }}_i).$$

(23)

The deformed q-entropic regularizer for an element ${\mathbf{\pi }}_i$ is $\Omega \left({\mathbf{\pi }}_i\right)=\frac{\lambda }{2-q}({\mathbf{\pi }}_i{log}_q\left({\mathbf{\pi }}_i\right)-{\mathbf{\pi }}_i)$.

The deformed q entropy recovers the Shannon entropy at the limit $q\nearrow 1$: $H_1\left(\mathbf{\pi }\right)=-{\sum }_i({\mathbf{\pi }}_{i}log\left({\mathbf{\pi }}_i\right)-{\mathbf{\pi }}_i)$. In addition, the limit $q\searrow 0$ recovers the negative of the squared 2-norm: $H_0\left(\mathbf{\pi }\right)=-\frac{1}{2}{\sum }_i({\mathbf{\pi }}_i^2-2{\mathbf{\pi }}_i)=-\frac{1}{2}{\parallel \mathbf{\pi }\parallel }_2^2+1$. Therefore, the deformed q entropy is an interpolation between the Shannon entropy and squared 2-norm. Hereafter, we consider the regularized OT with the deformed q entropy

$${\mathcal{T}}_{-\lambda H_q}(\mu ,\nu )=\underset{\mathbf{\Pi }\in \mathcal{U}(\mu ,\nu )}{inf}〈\mathbf{D},\mathbf{\Pi }〉-\lambda H_q(\mathbf{\Pi }),$$

(24)

by solving its dual counterpart. The deformed q entropy is different from the Tsallis entropy $T_q$ (see Equation (19)) in that the Tsallis entropy and deformed q entropy are defined by the q expectation$〈{\mathbf{\pi }}^q,·〉$ [34] and the usual expectation $〈\mathbf{\pi },·〉$, respectively, while both are defined by the q logarithm.

Remark 1.

The primary reason we picked the deformed q entropy $H_q$ to design the regularizer is owing to its natural connection to the q-Gibbs kernel through the dual map, $\nabla {(-\lambda H_q)}^{\star }\left(\eta \right)={exp}_q(\eta /\lambda )$. When the Tsallis entropy $T_q$ is used, the dual map is

$$\nabla {(-\lambda T_q)}^{\star }\left(\eta \right)=\frac{q^{1/(1-q)}}{{exp}_q(-\eta /\lambda )},$$

(25)

which is not naturally connected to the q-Gibbs kernel.  Muzellec et al. [35] proposed regularized OT with the Tsallis entropy, but they did not discuss its sparsity. As we show in Appendix D.1, the Tsallis entropy does not empirically induce sparsity.

In Figure 2, the deformed q entropy with a different deformation parameter is plotted for the one-dimensional simplex ${▵}^1$. One can easily confirm that $H_q\left(\mathbf{\pi }\right)$ is concave for $\mathbf{\pi }\in {\mathbb{R}}_{\ge 0}^n$, as illustrated in the figure.

4. Optimization and Convergence Analysis

4.1. Optimization Algorithm

We occasionally write $\Omega =-\lambda H_q$ to simplify the notation in this section. By simple algebra, we confirm

$${\Omega }^{\star }\left(\eta \right)=\frac{\lambda }{2-q}{exp}_q{\left(\frac{\eta }{\lambda }\right)}^{2-q},$$

(26)

which is convex because of the concavity of $H_q$. To solve Equation (24), we solve the dual

$${\mathcal{T}}_{-\lambda H_q}(\mu ,\nu )=\underset{\mathbf{\alpha }\in {\mathbb{R}}^n,\mathbf{\beta }\in {\mathbb{R}}^m}{sup}\underset{:=-\mathcal{F}\left(\mathbf{z}\right)}{\underbrace{-〈\mathbf{a},\mathbf{\alpha }〉-〈\mathbf{b},\mathbf{\beta }〉-\frac{\lambda }{2-q}\sum _{i,j}{exp}_q{\left(-\frac{{\mathbf{D}}_{ij}+{\mathbf{\alpha }}_i+{\mathbf{\beta }}_j}{\lambda }\right)}^{2-q}}},$$

(27)

where $\mathbf{z}:=(\mathbf{\alpha },\mathbf{\beta })$ denotes dual variables. As Equation (27) is an unconstrained optimization problem, many famous optimization solvers can be used to solve it; here, we use the BFGS method [24]. For the sake of convergence analysis (Section 4.2), we optimize the convex ${\ell }_2$-regularized dual objective

$$\begin{array}{cc}\hfill \mathrm{minimize}& \tilde{\mathcal{F}}\left(\mathbf{z}\right):=〈\mathbf{a},\mathbf{\alpha }〉+〈\mathbf{b},\mathbf{\beta }〉+\sum _{i,j}{\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j)+\frac{\kappa }{2}{\parallel \mathbf{z}\parallel }_2^2,\hfill \end{array}$$

(28)

where $\kappa >0$ represents the ${\ell }_2$-regularization parameter. In practice, ${\ell }_2$ regularization hardly affects the performance when $\kappa$ is sufficiently small. We can characterize the convergence rate by introducing (small) ${\ell }_2$ regularization, which makes the objective strongly convex, whereas the convergence guarantee without its rate is still possible without ${\ell }_2$ regularization [36].

We briefly summarize the algorithm in Algorithm 1, where ${\mathbf{d}}^{\left(k\right)}$, ${\rho }^{\left(k\right)}$, and ${\mathbf{g}}^{\left(k\right)}:=\nabla \tilde{\mathcal{F}}\left({\mathbf{z}}^{\left(k\right)}\right)$ represent the kth update direction, kth step size, and gradient at the current variable ${\mathbf{z}}^{\left(k\right)}$, respectively.

$${\mathbf{s}}^{\left(k\right)}:={\mathbf{z}}^{(k+1)}-{\mathbf{z}}^{\left(k\right)}\mathrm{and}{\mathbf{\zeta }}^{\left(k\right)}:={\mathbf{g}}^{(k+1)}-{\mathbf{g}}^{\left(k\right)}$$

(29)

are the differences of the dual variables and gradients between the next and current steps, respectively. Furthermore, let $(\gamma ,{\gamma }^{\prime })$ be the tolerance parameter for the Wolfe conditions, i.e., update directions and step sizes satisfy the conditions

$$\begin{array}{cccc}\hfill \tilde{\mathcal{F}}({\mathbf{z}}^{\left(k\right)}+{\rho }^{\left(k\right)}{\mathbf{d}}^{\left(k\right)})& \le \tilde{\mathcal{F}}({\mathbf{z}}^{\left(k\right)})+{\gamma }^{\prime }{\rho }^{\left(k\right)}{\mathbf{g}}^{\left(k\right)\top }{\mathbf{d}}^{\left(k\right)},\hfill & \hfill & (\mathrm{Armijo}\mathrm{condition})\hfill \end{array}$$

(30)

$$\begin{array}{cccc}\hfill {\mathbf{g}}^{(k+1)\top }{\mathbf{d}}^{\left(k\right)}& \ge \gamma {\mathbf{g}}^{\left(k\right)\top }{\mathbf{d}}^{\left(k\right)}.\hfill & \hfill & (\mathrm{curvature}\mathrm{condition})\hfill \end{array}$$

(31)

After obtaining the dual solution $(\widehat{\mathbf{\alpha }},\widehat{\mathbf{\beta }})$, the primal solution can be recovered from Equation (15).

4.2. Convergence Analysis

We provide a convergence guarantee for Algorithm 1. A technical assumption is stated beforehand.

Assumption 1.

Let ${\mathbf{z}}_{\star }$ be the global optimum of $\tilde{\mathcal{F}}$. For $\tau \in (0,1)$, we define the set ${\mathcal{Z}}_{\tau }\subseteq ridom\left(\tilde{\mathcal{F}}\right)$ as

$${\mathcal{Z}}_{\tau }:=\left\{\mathbf{z}|\nabla {\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j)\le \tau \mathrm{for}\mathrm{all}i,j\right\}.$$

(32)

Assume that ${\mathbf{z}}^{\left(K\right)}$ obtained by Algorithm 1 and ${\mathbf{z}}_{\star }$ are contained in ${\mathcal{Z}}_{\tau }$.

The dual map $\nabla {\Omega }^{\star }$ translates dual variables into primal variables, as in Equation (15). It is easy to confirm that ${\mathcal{Z}}_{\tau }$ is a closed convex set attributed to the convexity of $\nabla {\Omega }^{\star }$. Assumption 1 essentially assumes that all elements of the primal matrix (of ${\mathbf{z}}^{\left(K\right)}$ and ${\mathbf{z}}_{\star }$) are strictly less than 1; this always holds for ${\mathbf{z}}_{\star }$ (unless $n=m=1$) because of the strong duality. Moreover, this assumption is natural for ${\mathbf{z}}^{\left(K\right)}$ values sufficiently close to the optimum ${\mathbf{z}}_{\star }$. The bound parameter $\tau$ is a key element for characterizing the convergence speed.

Theorem 1.

Let $N:=max\{n,m\}$. Under Assumption 1, Algorithm 1 with the parameter choice $\kappa =2N{\tau }^q{\lambda }^{-1}$ returns a point ${\mathbf{z}}^{\left(k\right)}$ satisfying

$$\parallel {\mathbf{g}}^{\left(K\right)}{\parallel }_2<\sqrt{\frac{16(\tilde{\mathcal{F}}\left({\mathbf{z}}^{\left(0\right)}\right)-{\tilde{\mathcal{F}}}_{\star })N{\tau }^q}{\lambda }}{r}^K$$

(33)

where ${\tilde{\mathcal{F}}}_{\star }:={inf}_{\mathbf{z}}\tilde{\mathcal{F}}\left(\mathbf{z}\right)$ represents the optimal value of the ${\ell }_2$-regularized dual objective and $0<r<1$ is an absolute constant independent from $(\lambda ,\tau ,q,N)$.

The proof is shown in Section 4.3. We conclude that a larger deformation parameter q yields better convergence because the coefficient in Equation (33) is $O\left({\tau }^{q/2}\right)$ with the base $\sqrt{\tau }<1$. Therefore, the deformation parameter introduces a new trade-off: $q\searrow 0$ yields a more sparse solution but slows down the convergence, whereas $q\nearrow 1$ ameliorates the convergence while sacrificing sparsity. One may obtain the solution faster than the squared 2-norm regularizer used in Blondel et al. [20], which corresponds to the case $q=0$, by modulating the deformation parameter q.

In regularized OT, it is a common approach to use weaker regularization (i.e., a smaller $\lambda$) to obtain a solution sparser and closer to the unregularized solution; however, a smaller $\lambda$ results in numerical instability and slow computation [37]. This can be observed from Equation (33) because a smaller $\lambda$ drives its upper bound considerably large.

Subsequently, we compared the computational complexity of q-DOT with the BFGS method and Sinkhorn algorithm. Altschuler et al. [25] showed that the Sinkhorn algorithm satisfies coupling constraints within the ${\ell }_1$ error $\epsilon$ in $O\left(N^2(logN){\epsilon }^{-3}\right)$ time, which is the sublinear convergence rate. In contrast, our convergence rate in Equation (33) is translated into the iteration complexity $K=O(log\left(N{\epsilon }^{-1}\right))$, where $\parallel {\mathbf{g}}^{\left(K\right)}{\parallel }_2\le \epsilon$. The gradient of $\tilde{\mathcal{F}}$ is

$$\nabla \tilde{\mathcal{F}}\left(\mathbf{z}\right)=\left[\begin{array}{c}⋮\\ {\mathbf{a}}_i-\sum _{j=1}^m\nabla {\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j)+\kappa {\mathbf{\alpha }}_i\\ ⋮\\ {\mathbf{b}}_i-\sum _{i=1}^n\nabla {\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j)+\kappa {\mathbf{\beta }}_j\\ ⋮\end{array}\right],$$

(34)

and $\nabla {\Omega }^{\star }(·)$ represents the mapping from the dual variables $({\mathbf{\alpha }}_i,{\mathbf{\beta }}_j)$ to the primal transport matrix ${\mathbf{\Pi }}_{ij}$ in Equation (15). Therefore, the gradient norm of $\mathcal{F}$ and coupling constraint error are comparable when the ${\ell }_2$-regularization parameter $\kappa$ is sufficiently small. The overall computational complexity is $O(N^{2}log\left(N{\epsilon }^{-1}\right))$ because the one step of Algorithm 1 runs in $O\left(N^2\right)$ time; this is the linear convergence rate. To confirm the one step of Algorithm 1 runs in $O\left(N^2\right)$ time, we note that the update direction can be computed with $O\left(N^2\right)$ time by using the Sherman–Morrison formula to invert ${\mathbf{B}}^{\left(k\right)}$. In addition, the Hessian estimate can be updated with $O\left(N^2\right)$ time because ${\mathbf{B}}^{\left(k\right)}$ is the rank-1 update and the computation of its inverse only requires the matrix-vector products of size N. Hence, Algorithm 1 exhibits better convergence in terms of the stopping criterion $\epsilon$. The comparison is summarized in

Table 2. Comparison of the computational complexity of the Sinkhorn algorithm and deformed q-optimal transport. $N=max\left\{n,m\right\}$.

Sinkhorn	q-DOT
$O\left(N^2(logN){\epsilon }^{-3}\right)$	$O(N^{2}log\left(N{\epsilon }^{-1}\right))$

.

4.3. Proofs

To prove Theorem 1, we leveraged several lemmas shown below. Lemma 2 is based on Powell [24] and Byrd et al. [36]. The missing proofs are provided in Appendix C.

Lemma 1.

For the initial point ${\mathbf{z}}^{\left(0\right)}$ and sequence ${\mathbf{z}}^{\left(1\right)},{\mathbf{z}}^{\left(2\right)},\dots ,{\mathbf{z}}^{\left(K\right)}$ obtained by Algorithm 1, we define the following set and its bound:

$$\mathcal{Z}:=conv\left(\left\{{\mathbf{z}}^{\left(0\right)},{\mathbf{z}}^{\left(1\right)},{\mathbf{z}}^{\left(2\right)},\dots ,{\mathbf{z}}^{\left(K\right)}\right\}\right),R:=\underset{\mathbf{z}\in \mathcal{Z}}{sup}\underset{i,j}{max}\nabla {\Omega }^{\star }(-{\mathbf{D}}_{ij}-{\mathbf{\alpha }}_i-{\mathbf{\beta }}_j),$$

(35)

where $conv\left(S\right)$ represents the convex hull of the set S. Then, $\tilde{\mathcal{F}}:{\mathbb{R}}^{n+m}\to \mathbb{R}$ is $M_1$ strongly convex and $M_2$-smooth over $\mathcal{Z}$, where $M_1=\kappa$ and $M_2\le \kappa +2N{R}^q{\lambda }^{-1}$. Moreover, $\tilde{\mathcal{F}}$ is $M_2^{\prime }$-smooth over ${\mathcal{Z}}_{\tau }$ (defined in Equation (32)), where $M_2^{\prime }\le \kappa +2N{\tau }^q{\lambda }^{-1}$.

Lemma 2.

Let ${\mathbf{z}}^{\left(1\right)},{\mathbf{z}}^{\left(2\right)},\dots ,{\mathbf{z}}^{\left(k\right)}$ be a sequence generated by Algorithm 1 given an initial point ${\mathbf{z}}^{\left(0\right)}$. In addition, let $c_1$, $c_2$, $c_3$, $c_4$, and $c_5$ be the constants

$$\begin{array}{cccc}\hfill c_1& :=\frac{1-\gamma }{M_2},\hfill & \hfill & c_2:=\frac{n+m}{K}+M_2,\hfill \\ \hfill c_3& :={\left(\frac{K}{n+m}\right)}^{(n+m)/K}{c}_2^{\frac{n+m+K}{K}},\hfill & \hfill & c_4:=\frac{c_3}{1-\gamma },\hfill \\ \hfill c_5& :=\frac{2(1-{\gamma }^{\prime })}{M_1}.\hfill \end{array}$$

(36)

Then,

$$\tilde{\mathcal{F}}({\mathbf{z}}^{\left(K\right)})-{\tilde{\mathcal{F}}}_{\star }\le {\left(1-\frac{{\gamma }^{\prime }{c}_1{M}_1}{2c_4^2{c}_5^2}\right)}^{K/2}(\tilde{\mathcal{F}}({\mathbf{z}}^{\left(0\right)})-{\tilde{\mathcal{F}}}_{\star }).$$

(37)

Lemma 3.

Let $c_1$, $c_2$, $c_3$, $c_4$, and $c_5$ be the same constants defined in Lemma 2. Then,

$$\frac{{\gamma }^{\prime }{c}_1{M}_1}{c_4^2{c}_5^2}>\frac{{(1-\gamma )}^3{\gamma }^{\prime }{e}^{-2(n+m)/e}}{4{(1-{\gamma }^{\prime })}^2}{\left(\frac{M_1}{M_2}\right)}^3.$$

(38)

Proof of Theorem 1.

Because $\tilde{\mathcal{F}}$ is differentiable and strongly convex, there exists an optimum ${\mathbf{z}}_{\star }$ such that ${\mathbf{g}}_{\star }:=\nabla \tilde{\mathcal{F}}\left({\mathbf{z}}_{\star }\right)=\mathbf{0}$; this implies $\parallel {\mathbf{g}}^{\left(K\right)}{\parallel }_2={\parallel {\mathbf{g}}^{\left(K\right)}-{\mathbf{g}}_{\star }\parallel }_2$.

By using Assumption 1 and Lemma 1, we obtain $\parallel {\mathbf{g}}^{\left(K\right)}-{\mathbf{g}}_{\star }{\parallel }_2=\parallel \nabla \tilde{\mathcal{F}}\left({\mathbf{z}}^{\left(K\right)}\right)-\nabla \tilde{\mathcal{F}}\left({\mathbf{z}}_{\star }\right){\parallel }_2\le M_2^{\prime }{\parallel {\mathbf{z}}^{\left(K\right)}-{\mathbf{z}}_{\star }\parallel }_2$. In addition, $\parallel {\mathbf{z}}^{\left(K\right)}-{\mathbf{z}}_{\star }{\parallel }_2^2\le \frac{2}{M_1}(\tilde{\mathcal{F}}({\mathbf{z}}^{\left(K\right)})-{\tilde{\mathcal{F}}}_{\star })$ as $\tilde{\mathcal{F}}$ is $M_1$ strongly convex over $\mathcal{Z}$ and the stationary condition $\nabla \tilde{\mathcal{F}}\left({\mathbf{z}}_{\star }\right)=\mathbf{0}$ holds. We obtain the convergence bound by using Lemmas 2 and 3 as

$$\begin{array}{cc}\hfill \parallel {\mathbf{g}}^{\left(K\right)}{\parallel }_2& =\parallel {\mathbf{g}}^{\left(K\right)}-{\mathbf{g}}_{\star }{\parallel }_2\hfill \\ \hfill & \le M_2^{\prime }{\parallel {\mathbf{z}}^{\left(K\right)}-{\mathbf{z}}_{\star }\parallel }_2\hfill \\ \hfill & \le M_2^{\prime }\sqrt{\frac{2(\tilde{\mathcal{F}}\left({\mathbf{z}}^{\left(K\right)}\right)-{\tilde{\mathcal{F}}}_{\star })}{M_1}}\hfill \\ \hfill & \le M_2^{\prime }\sqrt{\frac{2(\tilde{\mathcal{F}}\left({\mathbf{z}}^{\left(0\right)}\right)-{\tilde{\mathcal{F}}}_{\star })}{M_1}{\left(1-\frac{{\gamma }^{\prime }{c}_1{M}_1}{2c_4^2{c}_5^2}\right)}^{K/2}}\hfill \\ \hfill & <M_2^{\prime }\sqrt{\frac{2(\tilde{\mathcal{F}}\left({\mathbf{z}}^{\left(0\right)}\right)-{\tilde{\mathcal{F}}}_{\star })}{M_1}{\left(1-\frac{{(1-\gamma )}^3{\gamma }^{\prime }{e}^{-2(n+m)/e}}{8{(1-{\gamma }^{\prime })}^2}{\left(\frac{M_1}{M_2}\right)}^3\right)}^{K/2}}\hfill \\ \hfill & \le \left(\kappa +\frac{2N{\tau }^q}{\lambda }\right)\sqrt{\frac{2(\tilde{\mathcal{F}}({\mathbf{z}}^{\left(0\right)})-{\tilde{\mathcal{F}}}_{\star })}{\kappa }{\left(1-\frac{C}{{(1+2N{R}^q{\lambda }^{-1}{\kappa }^{-1})}^3}\right)}^{K/2}},\hfill \end{array}$$

(39)

where we define $C:=\frac{{(1-\gamma )}^3{\gamma }^{\prime }{e}^{-2(n+m)/e}}{8{(1-{\gamma }^{\prime })}^2}$ and Lemma 1 is used at the last inequality to replace $M_1$, $M_2$ and $M_2^{\prime }$. We can immediately confirm $C\le \frac{1}{16}$ from $0<{\gamma }^{\prime }<\gamma <1$, ${\gamma }^{\prime }<\frac{1}{2}$, and $e^{-2(n+m)/e}<1$. Finally, by choosing $\kappa =2N{\tau }^q{\lambda }^{-1}$,

$$\begin{array}{cc}\hfill \parallel {\mathbf{g}}^{\left(K\right)}{\parallel }_2& \le \sqrt{\frac{16(\tilde{\mathcal{F}}\left({\mathbf{z}}^{\left(0\right)}\right)-{\tilde{\mathcal{F}}}_{\star })N{\tau }^q}{\lambda }{\left(1-\frac{C}{{(1+{(R/\tau )}^q)}^3}\right)}^{K/2}}\hfill \\ \hfill & \le \sqrt{\frac{16(\tilde{\mathcal{F}}\left({\mathbf{z}}^{\left(0\right)}\right)-{\tilde{\mathcal{F}}}_{\star })N{\tau }^q}{\lambda }}{r}^K,\hfill \end{array}$$

(40)

where we use ${(R/\tau )}^q\ge 1$ (owing to $R\ge \tau$ by definition) and let $r:={(1-C/8)}^{1/4}$ and $\sqrt[4]{127/128}\le r<1$. □

Remark 2.

More precisely, Altschuler et al. [25] showed that the Sinkhorn algorithm converges in $O\left(N^2{L}^3(logN){\epsilon }^{-3}\right)$ time, where ${L:=\parallel \mathbf{D}\parallel }_{\infty }$. For q-DOT, its computational complexity is not directly comparable to that of the Sinkhorn in L; instead, the following analysis provides us a qualitative comparison. First, the convergence rate of q-DOT in Equation (33) is translated into the iteration complexity $K=O(log\left(N{\epsilon }^{-1}\right)/log(1/r))$. The rate r is introduced in the proof of Theorem 1 (see Equation (40)): $r\ge {\left(1-\frac{C}{{(1+{(R/\tau )}^q)}^3}\right)}^{1/4}$. Then, by the Taylor expansion, we have a rough estimate $K\approx O(N^2{R}^{-3q}log\left(N{\epsilon }^{-1}\right))$, where R is a bound on the possible primal variables defined in Equation (35). We cannot directly compare $R^{-q}$ and L; nevertheless, $R^{-q}$ and L can be considered in the same magnitude given a reasonably sized domain $\mathcal{Z}$, noting that $\nabla \Omega \left(\pi \right)\approx O\left({\pi }^{1-q}\right)$. Hence, it is reasonable to suppose that both the Sinkhorn algorithm and q-DOT roughly converge in cubic time with respect to L.

5. Numerical Experiments

5.1. Sparsity

All the simulations described in this section were executed on a 2.7 GHz quad-core Intel^® Core^™ i7 processor. We used the following synthetic dataset: ${\left({\mathbf{x}}_i\right)}_{i=1}^n\sim \mathcal{N}({\mathbf{1}}_2,{\mathbf{I}}_2)$, ${\left({\mathbf{y}}_j\right)}_{j=1}^m\sim \mathcal{N}(-{\mathbf{1}}_2,{\mathbf{I}}_2)$, and $n=m=30$, where $\mathcal{N}(\mathbf{\mu },\mathbf{\Sigma })$ represents the Gaussian distribution with mean $\mathbf{\mu }$ and covariance $\mathbf{\Sigma }$. For each of the unregularized OTs, q-DOT, and Sinkhorn algorithm, we computed the transport matrices. For q-DOT and the Sinkhorn algorithm, different regularization parameters $\lambda$ were compared: $\lambda \in \left\{1\times {10}^{-2},1\times {10}^{-1},1\right\}$; and $\epsilon =1\times {10}^{-6}$ was used as the stopping criterion: q-DOT stopped after the gradient norm was less than $\epsilon$, and the Sinkhorn algorithm stopped after the ${\ell }_1$ error of the coupling constraints was less than $\epsilon$. We compared different deformation parameters $q\in \left\{0,0.25,0.5,0.75\right\}$ and fixed the dual ${\ell }_2$-regularization parameter $\kappa =1\times {10}^{-6}$ for q-DOT. The q-DOT with $q=0$ corresponded to a regularized OT with the squared 2-norm proposed by Blondel et al. [20]. For the unregularized OT, we used the implementation of the Python optimal transport package [38]. For q-DOT, we used the L-BFGS-B method (instead of the vanilla BFGS) provided by the SciPy package [39]. To determine zero entries in the transport matrix, we did not impose any positive threshold to disregard small values (as in Swanson et al. [6]) but regarded entries smaller than machine epsilon as zero.

The simulation results are shown in

Table 3. Comparison of the sparsity and cost with the synthetic dataset. Sparsity indicates the ratio of zero entries in each transport matrix. We counted the number of entries smaller than machine epsilon to measure the sparsity instead of imposing a small positive threshold for determining zero entries. Sinkhorn ($\lambda =0.01$) does not work well because of numerical instability.

	Sparsity	Cost $〈\mathbf{D},\widehat{\mathbf{\Pi }}〉$
Wasserstein (unregularized)	0.967	7.126
q-DOT ($q=0.00,\lambda =0.01$)	0.962	7.129
q-DOT ($q=0.00,\lambda =0.10$)	0.961	7.126
q-DOT ($q=0.00,\lambda =1.00$)	0.950	7.144
q-DOT ($q=0.25,\lambda =0.01$)	0.963	7.129
q-DOT ($q=0.25,\lambda =0.10$)	0.959	7.126
q-DOT ($q=0.25,\lambda =1.00$)	0.912	7.133
q-DOT ($q=0.50,\lambda =0.01$)	0.963	7.136
q-DOT ($q=0.50,\lambda =0.10$)	0.946	7.127
q-DOT ($q=0.50,\lambda =1.00$)	0.879	7.155
q-DOT ($q=0.75,\lambda =0.01$)	0.948	7.127
q-DOT ($q=0.75,\lambda =0.10$)	0.897	7.136
q-DOT ($q=0.75,\lambda =1.00$)	0.647	7.245
Sinkhorn ($\lambda =0.01$)	—	—
Sinkhorn ($\lambda =0.10$)	0.000	7.164
Sinkhorn ($\lambda =1.00$)	0.000	7.788

and Figure 3. First, we qualitatively evaluated each method by using Figure 3 such that q-DOT obtained a very similar transport matrix to the unregularized OT solution. The solution was slightly blurred with increases in q and $\lambda$. In contrast, the Sinkhorn algorithm output considerably uncertain transport matrices. Furthermore, the Sinkhorn algorithm was numerically unstable with a very small regularization such as $\lambda =0.01$.

From Table 3, we further quantitatively observed the behavior. The transport matrices obtained by q-DOT were very sparse in most cases, and the sparsity was close to that of the unregularized OT. Furthermore, we observed the tendency such that smaller q and $\lambda$ yielded a sparser solution. Significantly, the Sinkhorn algorithm obtained completely dense matrices (sparsity = 0). Although the transport matrices of q-DOT with $(q,\lambda )=(0.5,1),(0.75,1)$ appear somewhat similar to the Sinkhorn solutions in Figure 3, the former is much sparser. This suggests that a deformation parameter q slightly smaller than 1 is sufficient for q-DOT to output a sparse transport matrix.

For the obtained cost values $〈\mathbf{D},\widehat{\mathbf{\Pi }}〉$, we did not see a clear advantage of using a specific q and $\lambda$ from the results of q-DOT. Nevertheless, it is evident that q-DOT more accurately estimated the Wasserstein cost than the Sinkhorn algorithm regardless of the q and $\lambda$ used in this simulation.

5.2. Runtime Comparison

We compared the runtimes of q-DOT and the Sinkhorn algorithm using the same dataset as in Section 5.1, but with different dataset sizes: we chose $n=m\in \{100,300,500,$$1000\}$. The parameter choices were the same as in Section 5.1, except that the regularization parameter was fixed to $\lambda =0.1$. The result is shown in Figure 4; the larger deformation parameter q makes q-DOT converge faster when $n=m=100$. When $n=m\ge 300$, the difference between $q=0$, $q=0.25$, and $q=0.5$ was not as evident. This may be partly because we fixed the parameter choice $\kappa =1\times {10}^{-6}$ for the all experiments, unlike the oracle parameter choice $\kappa =2N{\tau }^q{\lambda }^{-1}$ (in Theorem 1) depending on q. Nonetheless, $q=0.75$ is clearly superior to the smaller q. From these observations, the trade-off between the sparsity and computation speed resulting from the deformation parameter q is theoretically established in Theorem 1 and it was empirically observed.

5.3. Approximation of 1-Wasserstein Distance

Finally, we compared the approximation errors of the 1-Wasserstein distance $|〈\mathbf{D},\widehat{\mathbf{\Pi }}〉-〈\mathbf{D},{\mathbf{\Pi }}_{♯}〉|$ of q-DOT and the Sinkhorn algorithm with different q and $\lambda$, where $\widehat{\mathbf{\Pi }}$ represents the computed transport matrix and ${\mathbf{\Pi }}_{♯}\in {arg\; min}_{\mathbf{\Pi }\in \mathcal{U}(\mu ,\nu )}〈\mathbf{D},\mathbf{\Pi }〉$ represents the LP solution. We used the same dataset and stopping criterion $\epsilon$ as described in Section 5.1 For the range of q, we used $q\in \left\{0.00,0.25,0.50,0.75\right\}$. For the range of $\lambda$, we used $\lambda \in \left\{0.05,0.1,0.5\right\}$.

The result is shown in Figure 5. The difference was not significant when q was small, such as $q\in \left\{0.00,0.25\right\}$. Once q became larger, such as $q\in \left\{0.50,0.75\right\}$, the approximation error evidently worsened. The Sinkhorn algorithm always exhibited worse approximation errors than q-DOT with q in the range used in this simulation regardless of $\lambda$. Formal guarantees for the 1-Wasserstein approximation error (such as Altschuler et al. [25] and Weed [40]) will be considered in future work.