Computing the Gromov-Wasserstein Distance between Two Surface Meshes Using Optimal Transport

Abstract

The Gromov-Wasserstein (GW) formalism can be seen as a generalization of the optimal transport (OT) formalism for comparing two distributions associated with different metric spaces. It is a quadratic optimization problem and solving it usually has computational costs that can rise sharply if the problem size exceeds a few hundred points. Recently fast techniques based on entropy regularization have being developed to solve an approximation of the GW problem quickly. There are issues, however, with the numerical convergence of those regularized approximations to the true GW solution. To circumvent those issues, we introduce a novel strategy to solve the discrete GW problem using methods taken from statistical physics. We build a temperature-dependent free energy function that reflects the GW problem’s constraints. To account for possible differences of scales between the two metric spaces, we introduce a scaling factor s in the definition of the energy. From the extremum of the free energy, we derive a mapping between the two probability measures that are being compared, as well as a distance between those measures. This distance is equal to the GW distance when the temperature goes to zero. The optimal scaling factor itself is obtained by minimizing the free energy with respect to s. We illustrate our approach on the problem of comparing shapes defined by unstructured triangulations of their surfaces. We use several synthetic and “real life” datasets. We demonstrate the accuracy and automaticity of our approach in non-rigid registration of shapes. We provide numerical evidence that there is a strong correlation between the GW distances computed from low-resolution, surface-based representations of proteins and the analogous distances computed from atomistic models of the same proteins.

1. Introduction

In 1776, Gaspard Monge presented an intriguing problem to the French Academy of Science [1]. Consider two domains $D_1$ and $D_2$ in the plane. $D_1$ (referred to as “déblai” by Monge) contains an excess of earth that needs to be transported to $D_2$ (“remblai” in Monge’s terminology). Assuming that the earth at a point $(x,y)$ in $D_1$ is transported to a position $F(x,y)$ in $D_2$, and that the masses associated with the point and image are equal, proportional to $dxdy$, the total “cost” C of the transportation is given by

$$\begin{array}{c}\hfill C=\int {\int }_{D_1}\left|\right|(x,y)-F(x,y)\left|\right|dxdy,\end{array}$$

(1)

where $\left|\right|.\left|\right|$ stands for the distance in the plane. Finding the minimum cost for moving the earth is then akin to finding this transport function F. Monge acknowledged in his presentation that he had not solved this problem on a practical level. This “allocation of ressources” problem became well known, however, reappearing in many disciplines and consequently has been the object of many studies. Erwin Schrödinger, for example, expressed it as the problem of finding how to evolve a probability distribution into another (see [2] for a review), the Schrödinger bridge problem. Kantorovich relaxed the Monge problem by allowing masses to split [3]. He also formulated a method, linear programming, for solving his relaxed version. All these problems are now referred to as the optimal transport problems, or OT. The OT problem is particularly intriguing since its solutions involve two crucial elements. It first specifies a distance between the measured spaces taken into account. This distance is known as the Monge-Kantorovich distance, the Wasserstein distance, or the earth mover’s distance, according to the field of applications. It also derives the optimal transport plan between the measured spaces, thereby defining a registration between the spaces. Consequently, applications of OT have exploded in the recent years (for in-depth reviews of OT and its uses, see [4,5]).

The Monge-Kantorovich OT problem can be formulated as follows. Let A and B be two subsets of a space M with a metric d, and let $\alpha$ and $\beta$ be probability measures on A and B, respectively. Let C be a cost function $C:A\times B\to {\mathbb{R}}^{+}$. The objective is to find a coupling G on $A\times B$ that minimizes a transportation cost U defined as

$$U\left(G\right)={\int }_A{\int }_{B}C(a,b)G(a,b)d(a,b).$$

(2)

The minimum of $U\left(G\right)$ is to be identified over the couplings G that satisfy the following constraints on their restrictions to the subsets A and B:

$$\begin{array}{cc}\hfill & \forall a,{\int }_{B}G(a,b)db=\alpha \left(a\right)\hfill \end{array}$$

(3a)

$$\begin{array}{cc}\hfill & \forall b,{\int }_{A}G(a,b)da=\beta \left(b\right).\hfill \end{array}$$

(3b)

It was shown that this minimum exists and that it defines a distance between the two probability measures $\alpha$ and $\beta$ that satisfies all metric properties [6].

One key condition for solving the optimal transport problem between two sets A and B is that those sets belong to the same metric space. This allows for the definition of a distance between any point of A and any point of B, and therefore of a cost function between the two sets. In practice, however, the two sets may not be in the same metric space, or, even if they do, the corresponding metric may not be practical. Consider for example two sets of points in ${\mathbb{R}}^3$; while it is possible to compute the euclidean distance between any pairs of points belonging to the two sets, inter distances between the two sets depend on the relative position of the two sets, namely a rigid body transformation involving six real-valued parameters (three that define a rotation, three that define a translation). Distances within each set are independent of such transformation; such distances, however, define a different metric space, one for each set of points. Several methods have been developed to use this information within the framework of optimal transport [7,8,9]. Here we are concerned with the Gromov-Wasserstein formalism [8], which has become popular for shape matching [8], for word embedding [10], as well as in the machine learning community, solving learning tasks such as heterogenous domain adaptation [11], deep metric alignment [12], computing distances between graphs [13] and graph classification [14,15], clustering [16] or generative modeling [17], among others. The GW problem can be stated as follows. Let $(A,d_A)$ and $(B,d_B)$ be two metric spaces, and let $\alpha$ and $\beta$ be probability measures on A and B, respectively. The goal is to find a coupling G on $A\times B$ that minimizes the transport cost T defined as

$$T_p\left(G\right)={\left({\int }_{A\times A}{\int }_{B\times B}G(a,b){\left|d_A(a,a^{\prime })-d_B(b,b^{\prime })\right|}^{p}G(a^{\prime },b^{\prime })dad{a}^{\prime }dbd{b}^{\prime }\right)}^{\frac{1}{p}},$$

(4)

with $p\ge 1$ (the most common value for p is 2, as it will be discussed below). The minimum of $T_p\left(G\right)$ is to be found over couplings that satisfy the constraints defined in Equation (3). As for the standard Monge-Kantorovich OT presented above, it was proved that a minimum to the transport cost defined in Equation (4) always exists [8]; we write this minimum as $G{W}_p(\alpha ,\beta )$. Finding this minimum, however, is a quadratic optimization problem, as opposed to finding the coupling that minimizes the transport cost associated with the OT problem (Equation (2)), which is a linear optimization problem. This minimum of the transportation cost defines a distance over the space of metric measured spaces (i.e., the triplet $(A,d_A,\alpha )$) modulo the measure-preserving isometries [8]. When the metrics $d_A$ and $d_B$ are the Euclidean distance over real numbers, i.e., when $A={\mathbb{R}}^m$ and $B={\mathbb{R}}^n$ (with n not necessarily equal to m), and $d_A={\left|\right|.\left|\right|}_{{\mathbb{R}}^m}$ and $d_B={\left|\right|.\left|\right|}_{{\mathbb{R}}^n}$ (where $\left|\right|.\left|\right|$ means the Euclidean norm), and when $p=2$, an interesting property of $G{W}_2(\alpha ,\beta )$ is that it is invariant with respect to rigid body transformations (i.e., isometries). This is of significance for example if the GW framework is to be used to compare shapes in space, when this comparison has to be independent of the relative positions of the shapes..

As mentioned above, classical OT is a linear programming problem. It is intriguing, however, that the current successes of OT are not due to recent advances in solving LP problems. They were instead prompted by the idea of entropic regularization, namely minimizing a modified version of the transport cost defined in Equation (2):

$$\begin{array}{ccc}\hfill U_{ϵ}\left(G\right)& =& U\left(G\right)-ϵH\left(G\right),\hfill \end{array}$$

(5)

where $ϵ$ is the parameter that controls the amount of regularization and $H\left(G\right)$ is an entropy on the coupling G. It is there to impose the positivity of its elements [18]. As $ϵ\to 0$ the regularized problem tends to the traditional problem. Interestingly, the minimum of the regularized transport cost $U_{ϵ}\left(G_{opt}\right)$ is a distance that satisfies all metric properties for all values of $ϵ$. This distance is called the Sinkhorn distance [18]. The main advantage of the entropic regularization is that it leads to a strictly convex problem that has a unique solution [18]. In addition, this solution can be found effectively using the Sinkhorn’s algorithm [19,20,21]. Sinkhorn’s algorithms have running times of order $O\left(N^2\right)$, while solving directly the OT problem as a linear program problem has a running time complexity of $O\left(N^3\right)$.

The same entropic regularization can be used the solve the GW problem, originally a quadratic optimization problem which is NP-hard in its general formulation. The idea is the same as for OT, namely add a regularized term to Equation (4) (see [22]):

$$T_{p,ϵ}\left(G\right)={\left(T_p\left(G\right)\right)}^p-ϵH\left(G\right).$$

(6)

Note that here it is not $T_p\left(G\right)$ that is considered, but its p-th power. While this simplifies the optimization process, it sets $T_{p,ϵ}\left(G\right)$ to be a “discrepancy” (in the language of Peyré et al. [22]) and not a distance with metric properties. The addition of the entropic regularization led, however, to an iterative algorithm for finding the GW discrepancy, with each iteration amounting to solve a regularized OT problem [22].

While the regularization based on entropy significantly expanded the appeal of OT, there are issues with the numerical convergence of the regularized solution to the actual OT solution. Furthermore, the physical significance of this regularization is unclear, despite its reference to entropy. Using methods from statistical physics, we have recently designed a novel framework for solving the OT problem that alleviate those issues [23,24]. The main idea is to build a strongly concave temperature dependent effective free energy function that encapsulates the constraints of the OT problem. The maximum of this function is proved to define a metric distance in the space of measured sets of points of fixed cardinality for all temperatures. In addition, this distance is proved to decrease monotonically to the regular OT distance at zero temperature. This property enables a robust algorithm for finding the OT distance using temperature annealing. This approach has been adapted to solving the assignment, or Monge problem [25], as well as to the unbalanced optimal transport problem [26]. In this paper we adapt it to solving the GW problem.

The paper is structured as follows. In the next section, we introduce the GW problem and its regularized version for discrete metric measured spaces. The following section covers the specifics of the statistical physics method we propose. All proofs of important properties of this method are given in the appendices. The subsequent section is devoted to the algorithm that implements our method in a C++ program, FreeGW. Next, we present numerical applications to the problem of comparing and registering 3D shapes, using examples based on synthetic data as well as on real data. The conclusion highlights possible future developments.

2. The Discrete Gromov-Wasserstein Problem

This section briefly describes the discrete Gromov Wasserstein transport problem and its regularized version. More thorough descriptions can be found in Refs. [7,8,22].

The discrete version of the GW problem is an optimal transport problem between two discrete probability measures whose supports are metric spaces $(M_1,d_1)$ and $(M_2,d_2)$ with possibly different metrics. Let $S_1$ and $S_2$ be subsets of $M_1$ and $M_2$ with cardinality $N_1$ and $N_2$, respectively. Each point k in $S_1$ (resp $S_2$) is characterized by a “mass” $m_1\left(k\right)$ (resp $m_2\left(k\right)$). We assume balance, namely that ${\displaystyle \sum _k}{m}_1\left(k\right)={\displaystyle \sum _l}{m}_2\left(l\right)$. In the following, these sums are set to 1, but the formalism could easily be adapted to handle a different values.

The discrete GW problem is defined as finding a coupling or transport plan G that minimizes the total transport cost U defined as

$$U_p\left(G\right)={\displaystyle \sum _{k,l}}{\displaystyle \sum _{k^{\prime },l^{\prime }}}G(k,l){\left|d_1(k,k^{\prime })-d_2(l,l^{\prime })\right|}^{p}G(k^{\prime },l^{\prime }),$$

(7)

where p is a fixed integer greater or equal to 1, and the summations extend over all $(k,k^{\prime })\in S_1^2$ and $(l,l^{\prime })\in S_2^2$. Note that G is a matrix of correspondence between points k in $S_1$ and points l in $S_2$. The minimization is to be performed over those matrices G that satisfy the following constraints

$$\begin{array}{cc}\hfill & \forall (k,l),G(k,l)\ge 0,\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill & \forall k,{\displaystyle \sum _l}G(k,l)=m_1\left(k\right),\hfill \end{array}$$

(8b)

$$\begin{array}{cc}\hfill & \forall l,{\displaystyle \sum _k}G(k,l)=m_2\left(l\right).\hfill \end{array}$$

(8c)

The set of all matrices G for which those conditions (8) are satisfied defines a polytope, which we refer to as $\mathcal{G}(S_1,S_2)$.

The minimization of the cost $U_p\left(G\right)$ yields an optimal transport plan $G_{opt}$. We refer to the minimum of the cost as $d_p(S_1,S_2)$. Note that $d_p(S_1,S_2)$ is not a distance. Its p-root, however, which we write as $G{W}_p(S_1,S_2)={\left(U_p\left(G_{opt}\right)\right)}^{\frac{1}{p}}$ is a metric distance between $(S_1,d_1)$ and $(S_2,d_2)$ quotiented by measure-preserving isometries [8].

Solving for the transport plan that minimizes Equation (7) under the constraints defined in Equation (8) is a non-convex quadratic optimization problem [27,28] and therefore $NP$-hard in the general case (see for example [29]). To circumvent this large computing cost when N is large, following Cuturi’s idea proposed for the optimal transport problem [18], Peyré et al. proposed a regularized version of Equation (7) [22]:

$$\begin{array}{ccc}\hfill U_{p,ϵ}\left(G\right)& =& {\displaystyle \sum _{k,l}}{\displaystyle \sum _{k^{\prime },l^{\prime }}}G(k,l){\left|d_1(k,k^{\prime })-d_2(l,l^{\prime })\right|}^{p}G(k^{\prime },l^{\prime })+ϵ{\displaystyle \sum _{k,l}}G(k,l)log\left(G(k,l)\right),\hfill \end{array}$$

(9)

where $ϵ$ is a parameter that controls the level of regularization. This parameter scales an entropic term, with the entropy set to $-xln\left(x\right)$, the standard information theory entropy, which imposes the positivity of the $G(k,l)$ terms [18]. This regularized GW problem can then be solved iteratively using a regularized linear optimal transport solver, as described in Algorithm 1.

Algorithm 1 is akin to a sequential quadratic programming method [30]. Briefly, given two sets of weighted points and the intra-set distances between those points, a “cost matrix” between the two sets is defined from the current transport plan G (initialized according to the masses of the points). This cost matrix is then used to solve a regularized linear optimal transport problem. The corresponding optimal transport plan is then used to update the cost matrix, and the procedure is then iterated until the transport plan does not change anymore (within a tolerance), i.e., when the transport plan and the cost matrix are consistent. There are many options to solve the regularized linear optimal transport problem in step 2, such as the Sinkhorn algorithm [20,21] initially proposed for solving the OT problem by Cuturi [18], or stabilized version of this algorithm [31,32,33,34,35], or using our own method based on statistical physics [23,24].

Algorithm 1 An iterative solver for the regularized GW problem.

Input: $N_1$ and $N_2$, the size of the two sets of points $S_1$ and $S_2$. The mass vectors $m_1\left(k\right)$ and $m_2\left(l\right)$, for $k\in [1,N_1]$ and $l\in [1,N_2]$. The distance matrices $d_1(k,k^{\prime })$ and $d_2(l,l^{\prime })$ over all points $(k,k^{\prime })\in S_1^2$ and $(l,l^{\prime })\in S_2^2$. Tolerance, TOL; regularization parameter, $ϵ$; N, maximum number of iterations. Initialize: Initialize transport plan $G_0(k,l)=m_1\left(k\right)m_2\left(l\right)$ for  $n=1,\dots ,N$  do     (1) Define “cost matrix” $C_n(k,l)={\displaystyle \sum _{k^{\prime }=1}^{N_1}\sum _{l^{\prime }=1}^{N_2}}{\left|d_1(k,k^{\prime })-d_2(l,l^{\prime })\right|}^p{G}_{n-1}(k^{\prime },l^{\prime })$.     (2) Solve $G_n={\displaystyle \underset{G}{\mathrm{argmin}}}U\left(G\right)={\displaystyle \underset{G}{\mathrm{argmin}}}\left({\displaystyle \sum _{k,l}}{C}_n(k,l)G(k,l)+ϵ{\displaystyle \sum _{k,l}}G(k,l)log\left(G(k,l)\right)\right)$ under the constraints defined in Equation (8).     (3) If $\left|\right|G_n-G_{n-1}\left|\right|<TOL$, break. end for Output: The optimal transport plan, $G_n$.

Algorithm 1 can be seen as applying successive linear approximation to the quadratic GW problem and as such it is expected to be efficient in computing time. There remains difficulties, however, as:

(i)

Solving the regularized OT problem in step 2 is difficult when $ϵ\to 0$ (a necessary condition to get to the real GW distance).

(ii)

Algorithm 1 is basically a fixed point method for which there is no guarantee of convergence. This is discussed in detail in Ref. [22].

(iii)

There is no easy option within Algorithm 1 to compute a scaling factor between distances within $S_1$ and distances within $S_2$. Those distances may have different scales, however, which can significantly impact the numerical stability of the algorithm.

In the following section, we describe a different method for solving the GW problem that attempt to solve at least some of these concerns.

3. A Statistical Physics Approach to Solving the Gromov-Wasserstein Problem

Solving the GW problem amounts to finding the minimum of a function defined by Equation (7) over the space of possible couplings between the two discrete sets of points considered. If this function is reworded as an “energy”, statistical physics allows for a different perspective on how to solve this problem. Indeed, in statistical physics, finding the minimum of an energy function is equivalent to finding the most probable state of the system it characterizes. Here, this system corresponds to the different couplings between between the measured sets of points $S_1$ and $S_2$. We refer to the space of such couplings $\mathcal{G}(S_1,S_2)$ (see above). Couplings G in this space satisfy multiple constraints. Their row sums and row columns correspond to the masses associated with $S_1$ and $S_2$, respectively (see Equation (8)). In addition, their elements are positive, and in fact smaller than one, if we assume that the sums of the masses on $S_1$ and on $S_2$ are both equal to 1 (the fact that these sums are equal is referred to as the balance condition and setting them to 1 is arbitrary but useful, as illustrated below).

A state in this system is then characterized with a coupling G and its energy value $U_p\left(G\right)$ as defined by Equation (7). To account for possible differences of scales between the metrics on $S_1$ and $S_2$, we introduce a scaling factor s between the distances $d_1$ and $d_2$:

$$U_p(s,G)=\sum _{k,l}\sum _{k^{\prime },l^{\prime }}G(k,l){\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}G(k^{\prime },l^{\prime }),$$

(10)

where p is a constant and s is considered as a variable. The probability of finding the system in a state characterized by G and s is:

$$P(s,G)=\frac{1}{Z_{\beta }(s,S_1,S_2)}{e}^{-\beta U_p(s,G)}.$$

(11)

In this equation, $\beta$ is the inverse of the temperature, namely $\beta =1/\left(k_{B}T\right)$ with $k_B$ the Boltzmann constant and T the temperature. $Z_{\beta }(s,S_1,S_2)$ is the partition function defined as

$$Z_{\beta }(s,S_1,S_2)={\int }_{G\in \mathcal{G}(S_1,S_2)}{e}^{-\beta U_p(s,G)}d{\mu }_{12}.$$

(12)

An interesting property of a partition function Z is that most thermodynamic variables of the system can be expressed as functions of Z, or as functions of its derivatives. This is the case for the free energy of the system:

$${\mathcal{F}}_{\beta }(s,S_1,S_2)=-\frac{1}{\beta }ln\left(Z_{\beta }(s,S_1,S_2)\right),$$

(13)

as well as for the average energy $E_{\beta }(s,S_1,S_2)=<U_p(s,G){>}_{s\in \mathbb{R},G\in \mathcal{G}}$ by

$$\begin{array}{c}\hfill E_{\beta }(s,S_1,S_2)=-\frac{\partial ln\left(Z_{\beta }(s,S_1,S_2)\right)}{\partial \beta }.\end{array}$$

(14)

In addition to finding the coupling matrix G, we want to find the scaling factor s that defines the best match between the two distributions. To reach this goal, we will minimize the free energy ${\mathcal{F}}_{\beta }(s,S_1,S_2)$ with respect to s. The minimized free energy is denoted by ${\mathcal{F}}_{\beta }(S_1,S_2)$ and similarly, the average energy computed with the minimal s is denoted as $E_{\beta }(S_1,S_2)$.

We start with an important property of the free energy and of the average energy:

Proposition 1.

For all $\beta >0$, the free energy and the average energy are monotonically decreasing functions of β. Both functions converge to $d_p(S_1,S_2)$ from which we can compute the GW distance as $G{W}_p(S_1,S_2)=d_p{(S_1,S_2)}^{\frac{1}{p}}$.

Proof. 

How the functions ${\mathcal{F}}_{\beta }$ and of $E_{\beta }$ behave as the parameter $\beta$ is increased is studied in Appendix A.    □

This approach to solving the GW problem is appealing. It is based on a temperature-dependent free energy with a monotonic dependence on the inverse of the temperature, $\beta$, and convergence to the actual GW distance at zero temperature. In practice, however, it is of limited interest because the partition function and thus the extrema of the free energy cannot be computed explicitly. We propose using the saddle point approximation to approximate these quantities. We will demonstrate that the corresponding mean field values have the same properties as the exact quantities defined above. These mean field values are easily calculated.

Following the method described in Ref. [23] to impose the constraints that define $\mathcal{G}(S_1,S_2)$, the partition function can be rewritten as

$$\begin{array}{ccc}\hfill Z_{\beta }(s,S_1,S_2)=& & {\int }_0^1\prod _{k,l}dG(k,l)e^{-\beta {\displaystyle \sum _{k,l}}{\displaystyle \sum _{k^{\prime },l^{\prime }}}G(k,l){\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}G(k^{\prime },l^{\prime })}\times \hfill \\ & & \prod _k\delta \left(\sum _{l}G(k,l)-m_1\left(k\right)\right)\prod _l\delta \left(\sum _{k}G(k,l)-m_2\left(l\right)\right).\hfill \end{array}$$

(15)

To account for the quadratic term in the exponential, we introduce new variables $C(k,l)$ that are constrained to mimic a cost function between $S_1$ and $S_2$:

$$\begin{array}{ccc}\hfill Z_{\beta }(s,S_1,S_2)=& & {\int }_0^1\prod _{k,l}dG(k,l){\int }_{-\infty }^{+\infty }\prod _{k,l}dC(k,l)\hfill \\ & & e^{-\beta {\displaystyle \sum _{k,l}}G(k,l)C(k,l)}\times \hfill \\ & & \prod _k\delta \left(\sum _{l}G(k,l)-m_1\left(k\right)\right)\prod _l\delta \left(\sum _{k}G(k,l)-m_2\left(l\right)\right)\hfill \\ & & \prod _{k,l}\delta \left(\sum _{k^{\prime },l^{\prime }}{\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}G(k^{\prime },l^{\prime })-C(k,l)\right).\hfill \end{array}$$

(16)

Using the Fourier representation of a delta function,

$$\begin{array}{c}\hfill \delta \left(x\right)=\frac{1}{2\pi }{\int }_{-\infty }^{+\infty }{e}^{-ixt}dt,\end{array}$$

(17)

the partition function can be recast with integrals only. To do so, we introduce the Fourier variables $D(k,l)$, $\lambda \left(k\right)$ and $\mu \left(l\right)$, with $(k,l)\in [1,N_1]\times [1,N_2]$. Omitting the normalization factors $1/\left(2\pi \right)$, the partition function can then be expressed as,

$$\begin{array}{ccc}& & Z_{\beta }(s,S_1,S_2)={\int }_{-\infty }^{+\infty }\prod _{k,l}dC(k,l){\int }_0^1\prod _{k,l}dG(k,l)e^{-\beta {\displaystyle \sum _{k,l}}C(k,l)G(k,l)}\times \hfill \\ & & {\int }_{-\infty }^{+\infty }\prod _{k}d\lambda \left(k\right)e^{-i\beta {\displaystyle \sum _{k,l}}\lambda \left(k\right)G(k,l)+i\beta \sum _k\lambda \left(k\right)m_1\left(k\right)}{\int }_{-\infty }^{+\infty }\prod _{l}d\mu \left(l\right)e^{-i\beta {\displaystyle \sum _{k,l}}\mu \left(l\right)G(k,l)+i\beta {\displaystyle \sum _l}\mu \left(l\right)m_2\left(l\right)}\hfill \\ & & {\int }_{-\infty }^{+\infty }\prod _{k,l}dD(k,l)e^{i\beta {\displaystyle \sum _{k,l}}D(k,l)C(k,l)-i\beta {\displaystyle \sum _{k,l}}D(k,l){\displaystyle \sum _{k^{\prime },l^{\prime }}}{\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}G(k^{\prime },l^{\prime })}.\hfill \end{array}$$

(18)

Note that we have introduced an explicit scaling factor $\beta$ for the variables $D(k,l)$, $\lambda \left(k\right)$ and $\mu \left(l\right)$. The corresponding terms are then consistent with the energy term. Note also that the terms within the integrals in Z are now complex functions, while the partition function Z itself is real. For sake of clarity, we include the i in $D(k,l)$, $\lambda \left(k\right)$ and $\mu \left(l\right)$, i.e., $D(k,l)\equiv iD(k,l)$, $\lambda \left(k\right)\equiv i\lambda \left(k\right)$ and $\mu \left(l\right)\equiv i\mu \left(l\right)$. Those variables are now complex.

After rearrangements,

$$\begin{array}{ccc}& & Z_{\beta }(s,S_1,S_2)={\int }_{-\infty }^{+\infty }dC(k,l){\int }_{-\infty }^{+\infty }dD(k,l){\int }_{-\infty }^{+\infty }{\displaystyle \prod _k}d\lambda \left(k\right){\int }_{-\infty }^{+\infty }{\displaystyle \prod _l}d\mu \left(l\right)\hfill \\ & & e^{\beta \left({\displaystyle \sum _{k,l}}C(k,l)D(k,l)+{\displaystyle \sum _k}\lambda \left(k\right)m_1\left(k\right)+{\displaystyle \sum _l}\mu \left(l\right)m_2\left(l\right)\right)}\hfill \\ & & {\int }_0^1{\displaystyle \prod _{k,l}}dG(k,l)e^{-\beta {\displaystyle \sum _{k,l}}G(k,l)(C(k,l)+\lambda \left(k\right)+\mu \left(l\right)+{\displaystyle \sum _{k^{\prime },l^{\prime }}}{\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}D(k^{\prime },l^{\prime }))}.\hfill \end{array}$$

(19)

Shifting $C(k,l)\leftarrow C(k,l)+{\displaystyle \sum _{k^{\prime },l^{\prime }}}{\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}D(k^{\prime },l^{\prime })$,

$$\begin{array}{ccc}& & Z_{\beta }(s,S_1,S_2)={\int }_{-\infty }^{+\infty }dC(k,l){\int }_{-\infty }^{+\infty }dD(k,l){\int }_{-\infty }^{+\infty }{\displaystyle \prod _k}d\lambda \left(k\right){\int }_{-\infty }^{+\infty }{\displaystyle \prod _l}d\mu \left(l\right)\hfill \\ & & e^{\beta \left({\displaystyle \sum _{k,l}}C(k,l)D(k,l)+{\displaystyle \sum _k}\lambda \left(k\right)m_1\left(k\right)+{\displaystyle \sum _l}\mu \left(l\right)m_2\left(l\right)\right)}{e}^{\beta \left({\displaystyle \sum _{k,l}}{\displaystyle \sum _{k^{\prime },l^{\prime }}}D(k,l){\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}D(k^{\prime },l^{\prime })\right)}\hfill \\ & & {\int }_0^1{\displaystyle \prod _{k,l}}dG(k,l)e^{-\beta {\displaystyle \sum _{k,l}}G(k,l)(C(k,l)+\lambda \left(k\right)+\mu \left(l\right))}.\hfill \end{array}$$

(20)

We can now perform the integration over the real variables $G(k,l)$ to get

$$\begin{array}{ccc}& & Z_{\beta }(s,S_1,S_2)={\int }_{-\infty }^{+\infty }dC(k,l){\int }_{-\infty }^{+\infty }dD(k,l){\int }_{-\infty }^{+\infty }{\displaystyle \prod _k}d\lambda \left(k\right){\int }_{-\infty }^{+\infty }{\displaystyle \prod _l}d\mu \left(l\right)\hfill \\ & & e^{\beta \left({\displaystyle \sum _{k,l}}C(k,l)D(k,l)+{\displaystyle \sum _k}\lambda \left(k\right)m_1\left(k\right)+{\displaystyle \sum _l}\mu \left(l\right)m_2\left(l\right)\right)}{e}^{\beta \left({\displaystyle \sum _{k,l}}{\displaystyle \sum _{k^{\prime },l^{\prime }}}D(k,l){\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}D(k^{\prime },l^{\prime })\right)}\hfill \\ & & {\displaystyle \prod _{k,l}}\frac{1-e^{-\beta \left(C(k,l)+\lambda \left(k\right)+\mu \left(l\right)\right)}}{\beta \left(C(k,l)+\lambda \left(k\right)+\mu \left(l\right)\right)}.\hfill \end{array}$$

(21)

We rewrite this partition function as

$$Z_{\beta }(s,S_1,S_2)={\int }_{-\infty }^{+\infty }dC(k,l){\int }_{-\infty }^{+\infty }dD(k,l){\int }_{-\infty }^{+\infty }{\displaystyle \prod _k}d\lambda \left(k\right){\int }_{-\infty }^{+\infty }{\displaystyle \prod _l}d\mu \left(l\right)e^{-\beta F_{\beta }},$$

(22)

where $F_{\beta }$ is the effective free energy defined by:

$$\begin{array}{ccc}\hfill F_{\beta }& & =-{\displaystyle \sum _{k,l}}D(k,l)C(k,l)-\left({\displaystyle \sum _k}\lambda \left(k\right)m_1\left(k\right)+{\displaystyle \sum _l}{\mu }_l{m}_2\left(l\right)\right)\hfill \\ & & +{\displaystyle \sum _{k,k^{\prime }}}{\displaystyle \sum _{l,l^{\prime }}}D(k,l){\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}D(k^{\prime },l^{\prime })\hfill \\ & & -\frac{1}{\beta }{\displaystyle \sum _{k,l}}ln\left(\frac{1-e^{-\beta \left(C\right(k,l)+\lambda (k)+\mu (l\left)\right)}}{\beta \left(C\right(k,l)+\lambda (k)+\mu (l\left)\right)}\right).\hfill \end{array}$$

(23)

Let $\overline{G}(k,l)$ and $\overline{s}$ be the expected values of $G(k,l)$ and s with respect to the probability function given in Equation (11) (i.e., the values that lead to this probability to be maximum). It is unfortunately not possible to compute these expected values directly as even though we now have an expression for the partition function, this expression is not analytical. We use instead the concept of a saddle point approximation (SPA). The SPA is computed by searching for the effective free energy extrema with respect to the variables $C(k,l)$, $D(k,l)$, $\lambda \left(k\right)$, $\mu \left(l\right)$, and s:

$$\begin{array}{c}\hfill \frac{\partial F_{\beta }}{\partial C(k,l)}=0\mathrm{and}\frac{\partial F_{\beta }}{\partial D(k,l)}=0,\\ \hfill \frac{\partial F_{\beta }}{\partial \lambda \left(k\right)}=0\mathrm{and}\frac{\partial F_{\beta }}{\partial \mu \left(l\right)}=0,\\ \hfill \frac{\partial F_{\beta }}{\partial s}=0.\end{array}$$

(24)

These equations define the following system of four equations:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}D(k,l)\hfill & =\varphi \left(\beta \right(C(k,l)+\lambda \left(k\right)+\mu \left(l\right)\left)\right)\hfill \\ C(k,l)\hfill & =2{\displaystyle \sum _{k^{\prime },l^{\prime }}}{\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}D(k^{\prime },l^{\prime })\hfill \\ {\displaystyle \sum _l}D(k,l)\hfill & =m_1\left(k\right)\hfill \\ {\displaystyle \sum _k}D(k,l)\hfill & =m_2\left(l\right)\hfill \end{array}\right.\end{array}$$

(25)

where,

$$\varphi \left(x\right)=\frac{e^{-x}}{e^{-x}-1}+\frac{1}{x}.$$

(26)

For all real values x, the function $\varphi \left(x\right)$ is defined and continuous (once we set $\varphi \left(0\right)=0.5)$. It is monotonically decreasing over $\mathbb{R}$, with the asymptotes $y=1$ and $y=0$ at $-\infty$ and $+\infty$, respectively.

The free energy $F_{\beta }(s,S_1,S_2)$ can then be minimized with respect to s, namely $\frac{\partial F_{\beta }}{\partial s}=0$, leading to the equation:

$$\begin{array}{c}\hfill {\displaystyle \sum _{k,k^{\prime }}}{\displaystyle \sum _{l,l^{\prime }}}D(k,l)d_2(l,l^{\prime }){\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p-2}(d_1(k,k^{\prime })-s·d_2(l,l^{\prime }))D(k^{\prime },l^{\prime })=0,\end{array}$$

(27)

which needs to be solved for s. Given $D(k,l)$, this equation is polynomial in s, with degree $p-1$. We will see in the implementation section that in the special case $p=2$, the solution is easy to obtain.

We have the following property that relates the solutions of the SPA system of equations to the expected values for the transport plan:

Proposition 2.

Let $\overline{S}$ be the expected state of the system with respect to the probability given in Equation (11). $\overline{S}$ is associated with an expected transport plan $\overline{G}$ and optimal scaling factor $\overline{s}$. Let $D^{MF}(k,l)$, $C^{MF}(k,l)$, ${\lambda }^{MF}\left(k\right)$, ${\mu }^{MF}\left(l\right)$ and $\overline{s}$ be the solutions of the system of Equations (25) and (27). Then the following identities hold,

$$\begin{array}{ccc}\hfill \overline{G}(k,l)& =& \varphi \left(\beta (C^{MF}(k,l)+{\lambda }^{MF}\left(k\right)+{\mu }^{MF}\left(l\right))\right)=D^{MF}(k,l).\hfill \end{array}$$

(28)

Note that the solutions are mean field solutions, hence the superscript $MF$.

Proof. 

See Appendix B.    □

Equation (28) shows that each element of $\overline{G}(k,l)$ is built to be in the range of $\varphi \left(x\right)$, namely $(0,1)$, as expected by the constraints on G. This optimal coupling matrix $\overline{G}(k,l)$ is real, and therefore the variables $C(k,l)$, $D(k,l)$, $\lambda \left(k\right)$ and $\mu \left(l\right)$ must be real. Otherwise stated, note that the integral defining the partition function (see Equation (22)) does not depend on the choice of the integration path. The saddle point Equation (25) shows that a path parallel to the real axis for each of the variables is preferred.

For a given value of $\beta$, the expected values $\overline{G}(k,l)$ define a coupling $G_{\beta }^{MF}=\overline{G}$ between $S_1$ and $S_2$ that is an extremum of the free energy defined in Equation (23). This extremum is referred to as $F_{\beta }^{MF}$ while the corresponding optimal internal energy is $U_{\beta }^{MF}={\displaystyle \sum _{k,k^{\prime }}}{\displaystyle \sum _{l,l^{\prime }}}{G}^{MF}(k,l){\left|d_1(k,k^{\prime })-\overline{s}·d_2(l,l^{\prime })\right|}^p{G}^{MF}(k^{\prime },l^{\prime })$. Those two values are mean field approximations of the exact free energy and internal energy defined in Equations (13) and (14), respectively. They satisfy the following properties:

Proposition 3.

$F_{\beta }^{MF}$ and $U_{\beta }^{MF}$ are monotonic decreasing functions of the parameter β. They both converge to the GW quantity $d_p(S_1,S_2)$, with the GW distance being ${\left(d_p(S_1,S_2)\right)}^{\frac{1}{p}}$.

Proof. 

See Appendix C.    □

The benefits of the proposed framework that recasts the GW problem as a temperature dependent process are visible from Proposition 3. First, because of the exponential ratio in the function $\varphi \left(x\right)$, the equations provide good numerical stability for computing the optimal coupling matrix G. Second, the energy associated with the solution of the modified problem approaches the traditional GW distance when $T\to 0$. Finally, the temperature-dependent convergence is monotonic.

4. Implementation

The preceding section defines a framework for solving the GW optimal transport problem for any value of the parameter p. In practice, most applications consider the square loss with $p=2$. This leads to two simplifications:

(i)

Faster computation of the “cost matrix” $\mathit{C}$.Recall that in the SPA system of equations, the cost matrix C is defined as:

$$\begin{array}{ccc}\hfill C(k,l)& =& 2{\displaystyle \sum _{k^{\prime },l^{\prime }}}{\left|d_1(k,k^{\prime })-s·d_2(l,l^{\prime })\right|}^{p}D(k^{\prime },l^{\prime }),\hfill \end{array}$$

with a total time complexity of $O\left(N_1^2{N}_2^2\right)$ to compute the whole matrix. In the special case $p=2$, the absolute value is not necessary and the equation can be rewritten in matrix form as

$$\begin{array}{c}\hfill C=2(d_1\odot d_1)D{\mathbb{1}}_{N_2}{\mathbb{1}}_{N_2}^T+2s^2{\mathbb{1}}_{N_1}{\mathbb{1}}_{N_1}^{T}D(d_2\odot d_2)-4s{d}_{1}D{d}_2,\end{array}$$

(29)

where ${\mathbb{1}}_N$ is a vector of ones of dimension N and ⊙ is the Hadamard product. The time complexity of computing C using this equation of $O(N_1^2{N}_2+N_1{N}_2^2)$, a significant improvement compared to the general case when $N_1$ and $N_2$ are large. This property was already proposed as “Proposition 1” by Peyré et al. [22].

(ii)

Computing the scaling factor$\mathit{s}$. In the general case, given the matrix D, solving Equation (27) for the scaling factor s amounts to finding the zeros of a polynomial function of degree $p-1$, with possibly $p-1$ real roots (see Equation (27)). In the specific case $p=2$, however, there is a unique solution to this problem, defined as

$$\begin{array}{c}\hfill s=\frac{{\displaystyle \sum _{k,k^{\prime }}}{\displaystyle \sum _{l,l^{\prime }}}D(k,l)d_1(k,k^{\prime })d_2(l,l^{\prime })D(k^{\prime },l^{\prime })}{{\displaystyle \sum _{k,k^{\prime }}}{\displaystyle \sum _{l,l^{\prime }}}D(k,l)d_2{(l,l^{\prime })}^{2}D(k^{\prime },l^{\prime })}.\end{array}$$

(30)

We have implemented the finite temperature GW framework for $p=2$ in a C++ program FreeGW that is succinctly described in Algorithm 2.

FreeGW is based on multiple iterative procedures. The outer loop performs a temperature annealing: the parameter $\beta$ (inverse of the temperature) is gradually increased. At each value of $\beta$, the scaling factor and transport plan are computed iteratively. First, they are initialized at their values at the previous temperature (step 3). A cost matrix is then computed (step 4) and a non linear system of equations defined by equations 3 and 4 of the SPA system (25) is solved using an iterative Newton-Raphson method (step 5). This step is akin to solving the optimal transport problem at this temperature. Complete details on how to solve this system can be found in Ref. [23,24]. Once this system is solved for $\lambda$ and $\mu$, a new estimate of the transport plan is derived (step 6). This new transport plan is then used to compute new estimates of the cost matrix (step 4) and of the scaling factor (step 8). The procedure is then iterated over both estimates. When these new estimates do not change anymore (within a tolerance TOL generally set to ${10}^{-4}$), the optimal coupling $G_{\beta }^{MF}$ and the associated energy $U^{MF}\left(\beta \right)$ are calculated. The program stops when the inverse of the temperature has reached its maximum value that was provided as input (usually ${\beta }_{inf}$ is set to ${10}^{+12}$.

Algorithm 2 FreeGW: a temperature dependent framework for computing the Gromov Wasserstein Distance between two weighted set of points belonging to two different metric spaces.

Input: $N_1$ and $N_2$, the size of the two sets of points $S_1$ and $S_2$. The mass vectors $m_1\left(k\right)$ and $m_2\left(l\right)$, for $k\in [1,N_1]$ and $l\in [1,N_2]$. The distance matrices $d_1(k,k^{\prime })$ and $d_2(l,l^{\prime })$ over all points $(k,k^{\prime })\in S_1^2$ and $(l,l^{\prime })\in S_2^2$. Initial and final temperatures ${\beta }_0$, ${\beta }_{inf}$. Tolerance, TOL; N, maximum number of iterations. Initialize: Initialize transport plan $G_0(k,l)=m_1\left(k\right)m_2\left(l\right)$ and scaling factor $f_0=1$; Set $STEP=\sqrt{10}$. Set ${\beta }^0={\beta }_0/STEP$ for$i=1,\dots$do     (1) Set ${\beta }^i=STEP\ast {\beta }^{i-1}$. If ${\beta }^i\ge {\beta }_{inf}$ break;     (2) Initialize $s_0=f_{i-1}$     for $j=1,\dots$ until convergence do         (3) Initialize $D_0=G_{i-1}$;         for $m=1,\dots$ until convergence do            (4) Compute $C_m$ from $D_{m-1}$ and $s_{j-1}$ using Equation (29).            (5) Solve non linear system of equations for $\lambda$ and $\mu$: $$\begin{array}{c}\hfill {\displaystyle \sum _l}\varphi \left({\beta }^i(C_m(k,l)+\lambda \left(k\right)+\mu \left(l\right))\right)=m_1\left(k\right)\\ {\displaystyle \sum _k}\varphi \left({\beta }^i(C_m(k,l)+\lambda \left(k\right)+\mu \left(l\right))\right)=m_2\left(l\right),\end{array}$$                   Set solutions as ${\lambda }^{sol},{\mu }^{sol}$            (6) Compute $D_m(k,l)=\varphi \left({\beta }^i(C_m(k,l)+{\lambda }^{sol}\left(k\right)+{\mu }^{sol}\left(l\right))\right)$            (7) If $\left|\right|D_m-D_{m-1}\left|\right|<TOL$, break         end for         (8) Compute current scaling factor $s_j$ using converged $D_m$ and Equation (30);         (9) If $|s_j-s_{j-1}|<TOL$, break     end for     (10) Update $G_i=D_m$ and $f_i=s_j$. end for Output: The converged transport plan $G_{{\beta }_{inf}}^{MF}=G_i$, the scaling factor $s_{{\beta }_{inf}}^{MF}=f_i$, and the corresponding $G{W}_2$ distance $\sqrt{U_{{\beta }_{inf}}^{MF}}$.

5. Computational Experiments

We present experimental results highlighting the advantages of using the GW framework to compare shapes defined by unstructured triangulations of their surfaces. We use synthetic (the TOSCA dataset) and “real life” (protein structures) datasets.

5.1. Shape Similarity: Synthetic Data from TOSCA

We use the Gromov-Wasserstein formalism to detect non-rigid shape similarity. The experiments were performed on meshes taken from the TOSCA non-rigid dataset [36,37]. Eleven classes of objects were considered (see Figure 1): 8 classes of animals, cats (9 poses), dogs (11 poses), gorilla (21 poses), horses (17 poses), seahorses (6 poses), shark (1 pose), wolves (3 poses), two male shapes, Michael (20 poses) and David (15 poses), one female shape, Victoria (24 poses), and one mythical shape, centaurs (6 poses), for a total of 133 shapes. Note that compared to the full TOSCA non rigid dataset, we removed all lions as their meshes had severe topological issues at the level of the mane. Each class consists of the same shape under different poses. These poses are the results of transformations that were designed to mimic non-rigid motions within objects (see [36,37], for details). Note that the different representatives within a class may be represented with different meshes (i.e., in addition to having different geometry those meshes may have different topologies), and may have different genera. Each shape is represented with a triangulated mesh with approximately 3400 vertices and 6600 faces, with the exception of the gorilla and seahorse meshes that include approximately 2100 vertices and 4200 faces. The Euclidean farthest point sampling procedure was used to select 1000 points from each shape’s set of vertices. In brief, one begins by selecting a point at random from the set of vertices. The second point is chosen among the remaining vertices as the one that is at the greatest distance from this first point. Subsequent points are always chosen to maximize the shortest distance to the previous points.

In each experiment, a pair of shapes i and j is represented with their sets of sampled vertices, $S_i$ and $S_j$, and the geodesic distance matrices between those vertices, $d_i$ and $d_j$. The geodesic distances are computed using the method proposed by Mitchell et al. [38] and implemented in the code “geodesic” by Danil Kirsanov, available at https://code.google.com/archive/p/geodesic/, and accessed on 1 June 2020. The masses associated to the vertices are set uniform, equal to $1/Nv$, where Nv is the number of vertices. The GW problem is solved using FreeGW up to convergence. At each value of $\beta$, $\sqrt{U_{\beta }^{MF}}$ (see above), defines $D{M}_{\beta }(i,j)$, i.e., the $(i,j)$-th element of the distance matrix $D{M}_{\beta }$ over all shapes in our TOSCA dataset. Note that $\sqrt{U_{\beta }^{MF}}$ satisfies the properties of a metric distance only when $\beta$ is large (at convergence with respect to $\beta$). We generated a set of distance matrices $D{M}_{\beta }$ for $\beta$ ranging from ${10}^5$ to ${10}^{12}$. Figure 1 provides graphical representations of $D{M}_{\beta }$ for two different temperatures, $\beta =3\times {10}^7$ and $\beta ={10}^{12}$. Note that the discrimination between the different shapes of TOSCA improves as $\beta$ increases.

In order to assess quantitatively how well the different distance matrices $D{M}_{\beta }$ classify correctly the shapes in TOSCA, we designed the following set of classification experiments. We first built a reference set: we selected randomly half the shapes from each of the 11 classes within TOSCA to form this reference set. Each remaining shape was then classified by considering its distances (derived from $D{M}_{\beta }$) to all shapes in the reference set, and assigning it to the class of the shape with the shortest such distance (this is a 1-nearest neighbor classification experiment). By comparing this predicted class with the actual class to which the shape belongs we derived an estimate for the probability of correct classification $P\left(\beta \right)$ based on $D{M}_{\beta }$. We repeated this procedure over 10,000 random selections of the reference set. In Figure 2, we plot the averaged $P\left(\beta \right)$ computed over those 10,000 experiments as a function of the inverse temperature $\beta$. The lower the temperature (or equivalently the higher the parameter $\beta$), the more discriminative the energy $U_{\beta }^{MF}$ is. The highest level of correct classification is already observed for $\beta ={10}^8$, i.e., significantly before convergence to the $G{W}_2$ distance, which is usually reached for $\beta >{10}^{11}$.

5.2. Shape Correspondence: Synthetic Data from SHREC19

The second test case we consider is shape correspondence: identifying corresponding points between two (or more) 3D shapes. Note that this is different from shape registration, namely finding a transformation that brings one shape “close” to another. Indeed, correspondence can be derived from registration, while the reverse may not be true. The Gromov-Wassertein framework allows for finding correspondence, as the latter is embedded in the optimal transport plan it computes.

To assess how well GW can recover correspondence, we considered the SHREC19 benchmark [39]. This benchmark includes 3D shapes represented with a triangular mesh of their surfaces. These shapes are derived from 3D scans of real-world objects, with each object being present in multiple poses associated with one or more types of deformation. The deformations are classified into four different groups, referred to as test-sets. Those four groups correspond to articulated deformations (group 0), isometric deformations (group 1), non-isometric deformations (group 2), and topologic/geometric deformations (group 3). Example of shapes for each test-set are provided in Figure 3.

The SHREC19 benchmark includes 76 shape pairs that are selected from the four different groups, and regrouped in four test sets (

Table 1. Quality of different methods for computing correspondence between 3D shapes.

Method a	Test-Set 0	Test-Set 1	Test-Set 2	Test-Set 3	All Test Sets
RPTS [40]	0.920 b	0.926	0.824	0.929	0.899
NRP [41]	0.878	0.899	0.801	0.858	0.862
WRAP	0.853	0.920	0.772	0.870	0.856
KM [42]	0.760	0.865	0.757	0.799	0.804
FreeGW d	0.706	0.879	0.550	0.320	0.588
Algo1 e	0.666	0.846	0.490	0.338	0.548
GISC [43]	0.565	0.659	0.674	NA c	NA c

^a Rows are in descending order of score over all test sets. ^b Area under the curve (AUC) for the cumulative distribution functions of the normalized geodesic errors in the correspondences. Results for the first five methods are derived from Ref. [39]. ^c Not Available in Ref. [39]. ^d Our results based on Algorithm 2 that consider an annealing procedure in the regularization parameters. ^e Our results based on Algorithm 1.

). Test-set 0 includes 14 pairs of articulating wooden hands from group 0. Test-set 1 includes 26 pairs of models corresponding to clothed humans as well as hands from group 1. Test-set 2 includes 19 pairs of models from the group 2. Each of those pairs includes a thin clothed mannequin and a larger mannequin, ensuring the the transformation is non-isometric. Finally test-set 3 includes 17 pairs of shapes from group 3 that contain challenging geometric and topological changes. We chose the low resolution version of this benchmark. In this version, each shape is represented by approximately 10,000 vertices and 20,000 triangles. For each pair of shapes, the ground-truth correspondence is known.

The quality of shape correspondence is evaluated by measuring normalized geodesics between the ground-truth (available as part of the SHREC19 dataset) and the predicted correspondence that is derived from the GW optimal coupling. Specifically, let $x_i$ be a point on shape X, $y_i$ its predicted correspondence on shape Y and $g_i$ the ground truth position of $x_i$ on Y. Note that $y_i$ and $g_i$ are both on the surface of Y. The normalized geodesic error $ϵ\left(x_i\right)$ between the $y_i$ and $g_i$ is computed as:

$$\begin{array}{c}\hfill ϵ\left(x_i\right)=\frac{d_Y(y_i,g_i)}{area{\left(Y\right)}^{1/2}},\end{array}$$

where $d_Y(y_i,g_i)$ is the geodesic distance between $y_i$ and $g_i$ on the surface of Y. The geodesic distance is computed with the algorithm from Mitchell et al. [38], as described in the previous subsection.

We compared two different implementations of the GW framework, based on two different algorithms, named the fixed point method described in Algorithm 1 and the physics-based algorithm described in Algorithm 2, both implemented in FreeGW. Algorithm 1 uses a fixed regularization parameter, $ϵ$. To make sure that the corresponding transport plan is close to the actual GW transport plan, we chose $ϵ={10}^{-12}$. Traditional regularized OT solver do not work well for such a small $ϵ$. We chose therefore our own OT solver [23,24] to solve step 2 in Algorithm 1. In contrast, Algorithm 2 is based on an annealing plan with respect to the parameter $\beta$, such that when $\beta$ is large, the outputs of the program are guaranteed to match the actual GW results.

In each experiment, a pair of shapes i and j is represented with their complete sets of vertices, $S_i$ and $S_j$, and the geodesic distance matrices between those vertices, $d_i$ and $d_j$. The masses of the vertices are set uniform. The GW problem is solved using the two algorithms mentioned above. Both derive a transport plan G. For each vertex x on i, its correspondence y on j is set to the index of the maximum value on the row corresponding to x in G. Figure 4 shows the corresponding cumulative geodesic errors for the two algorithms for the four test sets in SHREC19. We observed that Algorithm 2 as implemented in FreeGW performs better for shape correspondence than the simpler Algorithm 1, on all test sets, and significantly better from shapes with isometric deformations (test set 1).

The SHREC19 dataset was originally used as a benchmark for a competition on comparing 3D shape registration that was part of the Workshop on 3D Object Retrieval that was held in Genova, Italy in may 2019. Several groups entered in this competition; results were published in Ref. [39]. Here we compare results based on the GW framework, as presented above, with results from the five top methods that were part of this competition. We briefly describe those five methods below.

In the first method, dubbed RTPS, the correspondence map between the template shape and the target shape is computed iteratively. Each iteration is built from two successive steps. In the first step, the vertices estimated to be in correspondence between the template and target shapes are derived from the registration result obtained from the previous iteration. These computed correspondences are used to derive a correspondence mapping. In the second step, the mapping is updated by using the closest points between template and target shapes identified by the mapping to find additional points that are in correspondence [40]. The second method, NRPA, is also based on registration. It consists of four key components, namely modeling of the deformation (assumed to be anisotropic and non-isometric), computing the correspondences, pruning those correspondences, and optimizing the deformation [41]. The third method, KM, is a kernel-based method, where the registration problem is formulated as matching between a set of pairwise and pointwise descriptors, imposing a continuity prior on the mapping [42]. The fourth method, GISC, uses a genetic algorithm to find the permutation matrix that encodes the correspondences between the vertices of the two shapes to compare [43]. Note that this is the most similar method to the GW formalism, as this permutation matrix is akin to a transport plan. Finally the fifth method, WRAP, is based on the commercial software WARP that includes a wrapping tool that non-rigidly fits one 3D shape to another, from which correspondence can be derived.

We report the results of the comparisons of the qualities of the different methods in Table 1 as the areas under the curve (AUC) for the cumulative distribution functions of the geodesic normalized errors. Results are divided according to the test sets of SHREC19, as well as summarized over all test sets.

There are a few observations we can make based on Table 1. First, the AUC values provides a quantification of the quality of a method for finding correspondence: the larger the value, the better the method. In particular, Table 1 confirms that the correspondences computed from the GW framework with temperature annealing (Algorithm 2) are better than those computed with the fixed point method described in Algorithm 1. Second the four methods based on registration, RTPS, NRP, WRAP, and KM, all perform better than the method that only compute correspondences. This is likely due to the fact the the deformations included in the SHREC19 dataset are all based on a mathematical morphing, and therefore are expected to be captured with a mapping function. Finally, the GW formalism performs better than a genetic algorithm (implemented in GISC). This genetic algorithm is the one closest to the GW formalism in its concept.

5.3. Shape Similarity: Morphodynamics of Protein Structure Surfaces

Different experimental techniques lead to different representations of protein structures. For example, high-resolution X-ray crystallography and NMR techniques derive models of proteins that include all their atoms and that are accurate at the Angstrom level. Recent progress on cryo electron microscopy (EM) can now often also reach atomic resolution. However, the difficulty of maintaining the integrity of stable complexes of interest on the various types of grids necessary to mount the sample in thin ice makes it often desirable to resort to negative staining techniques, in which case the resolution is much lower (typically 10–25 Angstroms). A low-resolution model derived from EM techniques is often represented as a density map, namely a shape characterized by its surface. Such a model is often available long before its high-resolution counterpart, as EM techniques are usually easier, faster, and cheaper to implement. It is therefore of interest to develop methods that can analyze the geometry of a protein directly from its EM density map. Such methods should generate information similar to those derived from methods that work directly on the high-resolution model of the protein structure. We consider here the problem of comparing the geometry of two protein structures using the GW distance between the surfaces of their density maps. To assess if this distance obtained from low resolution models of the protein structures mimic what could be found from high-resolution models, we compared it with the cRMS distance between atomistic models of the same proteins. We performed these tests on the protein calmodulin. Results are shown in Figure 5.

Calmodulin is a calcium binding protein that is found in all eukaryotic cells. Its structure looks like a dumbbell, with two small domains separated by a linker region. It is the flexibility of this linker that defines the ability of calmodulin to bind to a wide range of ligands [45].

We considered two conformations for calmodulin, a conformation in the absence of a ligand (referred to as the apo or ligand-free conformation) and a conformation in the presence of a substrate (referred to as holo ligand-bound conformation, where we use interchangeably the terms ligand and substrate to indicate a molecule that binds to calmodulin). Those conformations were found in the database of protein structure, the PDB [46], with codes 1CLL and 1A29, respectively. We built a trajectory between these two conformations. This trajectory is designed to mimic the structural transition that results from the binding of the ligand. We used the program MinActionPath who is designed to generate the most probable trajectory between the two conformers, namely the one with minimal action (for details, see [44]). The trajectory was sampled over 51 conformations, each represented with all the atoms of calmodulin. We then computed the distances between any two of these conformations in two different ways. First, we used the coordinate Root Mean Square (cRMS) distance computed over the $C_{\alpha }$ atoms of the high-resolution structures (see Refs. [47,48] for details on how to compute the cRMS). Second, we compared the same structures using their skin surfaces [49]. To derive those skin surfaces, we started with the common convention in chemistry to represent a structure as a union of balls, with each ball corresponding to an atom. The coordinates of an atom define the center of a ball that is associated with it. The atom is also characterized with a van der Waals radius based on its chemical nature. The radius of the ball is then set to this vdW radius, plus a probe radius of $R=1.4$ Å, designed to mimic a water molecule in its proximity. The skin surface is then defined as the boundary of this union of balls. We generated a triangular mesh on the skin surface using the program smesh [50,51]. We found that those meshes have similar sizes for all 51 conformations we considered, with on average approximately 40,000 vertices and 70,000 triangles. 1000 points were selected from each mesh using the Euclidean farthest point sampling procedure described above for the TOSCA dataset. We compared these sampled meshes using FreeGW.

We compared all 51 conformations of calmodulin with both the apo and holo forms, using the cRMS and the GW distances. Results of these calculations are shown in Figure 5. We do observe that the GW distances measured based on the low resolution skin surfaces correlate well with the cRMS distances computed from the high resolution, atomistic representations of the proteins. The correlation have coefficients above 0.96.

5.4. How Round Is Calmodulin?

A visual inspection of Figure 5 indicates that the ligand-bound conformation of calmodulin is more compact that its ligand-free conformation. To quantify this idea of “compactness” we use two independent measures of the surface of the proteins:

(i)

The sphericity S of a surface F quantifies how well it encloses volume. It is expressed as the surface area of an equivalent sphere (i.e., with the same volume V as the volume enclosed by F) divided by the surface area A of F:

$$\begin{array}{c}\hfill S={\pi }^{1/3}{\left(6V\right)}^{2/3}/A.\end{array}$$

(31)

The sphericity is at most one, and equals one only for the round sphere,

(ii)

The GW distance between the surface of the protein and the surface of a round sphere.

We computed these two measures on all 51 conformations of the trajectory described in the previous subsection. Note that to compare the surfaces of the different conformations of calmodulin to the round sphere, we need a triangular mesh on the surface of that sphere. We generated this mesh by placing $N=1000$ points uniformly on the sphere and generating a triangulation from these points. We used the Matlab package “Uniform sampling of the sphere” available from [52] to position the points and QHull [53] to generate the triangulation.

We compared the 51 sampled meshes representing the 51 conformations of calmodulin (see above for details) with the mesh representing the surface of the sphere using FreeGW. Results of these calculations are compared to the corresponding sphericity of the meshes computed using Equation (31) in Figure 6. The GW distances and the sphericity are (anti) correlated (correlation coefficient: −0.8): as the sphericity increases, the level of correspondence between the protein surface and the sphere increases, and the GW distance decreases. We observe an inflection point for the sphericity along the trajectory at the 40th conformation: the GW distance shows a similar inflection point at the same conformation. This indicates that the GW distance between a protein represented with its surface and the sphere has value as a tool to assess the compactness of that protein.

6. Discussion

In the discrete Gromov Wasserstein problem, each set of points considered is characterized by a distance matrix that captures all pairwise distances between the points. When comparing two such sets of points, there is no guarantee that those distances have the same scale. For example, for the problem of comparing 3D shapes discussed in this paper, it is possible that those shapes were captured with different 3D scanners with different internal references. The shapes themselves may have different scales, for example when comparing animals of different sizes. One approach to circumvent this problem is to normalize the corresponding distance matrices, for example by setting the largest distance in each matrix to be 1. This approach is not optimal, especially in the presence of noise, as it is biased towards a single distance. While there are other ways to normalize a distance matrix, we have used a different approach to handle the scaling problem. Instead of arbitrarily scaling the distance matrices, we have added a free scaling parameter in our approach that is concurrently optimized with the transport plan between the two sets of points.

One of the current limitation of the algorithm we propose, Algorithm 2 implemented in FreeGW, is that it is demanding in computing power. It includes three nested loops: the outer loop controls the temperature annealing, the middle loop allows for an iterative update of the scaling factor, while the most inner loop is used to solve the SPA system at a given scaling factor by iterating over a cost matrix between the two sets of points. In addition, this SPA system is non linear and therefore it is also solved iteratively (see Refs. [23,24]). We have used a Newton-Raphson approach to solve this system. It should be noted that this approach requires that the Hessian of the free energy be computed (i.e., the Jacobian of the system of non linear equation). There are ways, however, to solve the system without the need to compute second derivatives (see for example Ref. [54]). We will try such alternate approaches in future work. For large problems, the overall computational cost can become large. For example, comparing two shapes of the SHREC19 benchmark, each with 10,000 points, require on average 12,500 s (i.e., approximately 3.5 h). There are several options to reduce this computing time. First, all calculations presented in this paper were run to convergence, i.e., up to an inverse temperature $\beta ={10}^{+12}$. As shown in Figure 2, if the problem is to classify shapes, there is no need to go to such a large value for $\beta$. Second, the size of the problem itself can be reduced by sampling: this is the approach we used for comparing shapes in the TOSCA dataset for example. Comparing two shapes of the TOSCA dataset, each with 1000 points, require on average 70 s. However, none of those solutions are general. For instance, sampling cannot be applied if we are interested directly in the transport plan between the two sets of points, and not just in the optimized distance between the sets. We will work on the problem of optimizing the running time of our algorithm in future studies.

7. Conclusions

In this study, we developed a novel method based on statistical physics for solving the discrete Gromov Wasserstein (GW) problem. Given two sets of measured points $S_1$ and $S_2$ associated with two possibly different metric spaces, the GW problem amounts to finding a correspondence between those points, stored in a transport plan, which minimizes an energy based on comparisons of pairwise distances within each set. We build a free energy function that, at a finite temperature, reflects the GW problem’s constraints. While the extremum of this free energy cannot be computed exactly, it can be estimated using a saddle point approximation. At each temperature, the corresponding mean field solution defines an optimal coupling between the two discrete probability measures that are compared, and a distance between those measures. We proved that this distance approaches the traditional GW distance when $T\to 0$ in a monotonic way, thereby amenable to temperature annealing. We have illustrated the usefulness of our approach on the problem of comparing shapes defined by unstructured triangulations of their surfaces and revealed that it allows for accurate and automatic non-rigid registration of shapes. We have shown that the GW distances computed from low-resolution, surface-based representations of proteins correlate well with the corresponding distances computed from atomistic models for the same proteins.

It is important to realize that the method we have proposed to solve the GW problem only applies under the assumption of balance, namely to problems in which the sum of the masses on the two discrete set of points are equal. This is often too restrictive in many applications, such as those in which only a partial mapping is sought out. The unbalanced GW problem is an open problem [55], which we intend to work on.