Computing Rate–Distortion Functions of Continuous Memoryless Sources via Discrete Algorithms: An Integrated Scheme with Convergence Guarantee and Algorithmic Acceleration

Abstract

Numerical computation of the rate–distortion (RD) function is a key problem in RD theory. Thus far, efficient algorithms have been well studied for discrete sources, but for continuous sources, there is still lack of a rigorously developed solution. In this article, an integrated approach is conducted that bridges RD problems of continuous memoryless sources and discrete numerical algorithms. First, we analyze and establish the theoretical convergence guarantee when progressively approaching the continuous RD problem via a sequence of discrete problems. Next, discrete algorithms including the Blahut–Arimoto (BA) and constrained BA algorithms are reviewed, and estimates of their required amount of arithmetic operations to attain $\epsilon$-accuracy in solving the continuous RD problem are derived. Finally, acceleration techniques that exploit structures of special distortion measures (i.e., squared-error and absolute-error) are developed.

1. Introduction

Rate–distortion (RD) theory traces back to Shannon’s seminal work on lossy source coding, where the RD function was introduced to characterize the fundamental trade-off between rate and distortion for representing a source subject to a fidelity criterion [1]. RD theory has been instrumental to the development of signal quantization and compression [2], and has long-lasting impacts on standards for media compression [3,4,5]. See [6] for a systematic treatment of standard and classical aspects of RD theory, [7] for a survey on the development of RD theory, [8] for introduction on RD theory in distributed settings, and [9] for recent progresses on the interplay between RD theory and machine learning.

Evaluating the RD function is a key problem in rate–distortion theory, and due to the lack of closed-form expressions in general settings, numerical algorithms are usually necessary. Thus far, efficient algorithms have been well studied for discrete sources, including the famous Blahut–Arimoto (BA) algorithm [10,11] and several others such as the constrained BA (CBA) algorithm [12] and expectation–maximization (EM)-type algorithms utilizing Bregman projections or mirror descent [13]. But for continuous sources, there is still lack of a rigorously developed solution. A commonly adopted approach is first discretizing the source and reproduction spaces and then solving the resulting discrete RD problem. Do solutions of the discretized problems provably converge to those of the original continuous problem? This basic question has remained unanswered to the best of our knowledge. It should be noted that, due to the nonlinear optimization formulation of the RD problem, naive discretization does not necessarily yield a convergent sequence of solutions towards those of the original continuous problem [14].

This article conducts an integrated approach for the computation of RD functions, bridging continuous RD problems and discrete numerical algorithms. Specifically, we (i) for continuous RD problems, construct suitable discretizations and prove that their solutions converge to those of the original continuous problem; (ii) review, based on [12], estimates of the required amount of arithmetic operations when applying the BA and CBA algorithms for attaining $\epsilon$-accuracy within the solution of the continuous RD problem; and (iii) design acceleration techniques that substantially reduce time and memory costs for squared-error and absolute-error distortion measures. A preliminary result pertaining to (i) was first presented in part at the IEEE International Symposium on Information Theory (ISIT) in 2024 [15], while here we present the complete work with proofs.

In our approach, we analyze a suitable dual form of the RD function, constructing a sequence of finite-dimensional spaces that progressively approximate the infinite-dimensional space of reproduction distributions. The convergence of discrete problem solutions to the continuous one is established by using standard tools in mathematical analysis [16]. The proof is algorithm-agnostic and does not depend on specific numerical algorithms used. In an alternative approach, mapping-based methods jointly optimize locations and masses of the discrete reproduction distribution [17], but the explicit dependence on locations makes the optimization problem nonconvex and typically lacks uniform performance guarantees [18]. By avoiding optimization over locations and instead working in the limit of asymptotically vanishing discretization step size, our approach provably guarantees convergence of both objective values and solution sequences. In another work [19], the RD function under a fixed output alphabet size constraint is evaluated, and corresponding encoding schemes are proposed.

With convergence guarantees, computing the continuous RD function can be effectively reduced to solving the discretized RD problems with numerical algorithms, i.e., the BA or CBA algorithm. To assess the efficiency of this approach, we review the arithmetic complexity required to attain an $\epsilon$-accurate solution. According to the analysis in [12], for the CBA algorithm, this complexity is $O\left({\displaystyle \frac{\left|\log \epsilon \right|}{{\epsilon }^{d+1}}}(1+\log |\log \epsilon \left|\right)\right)$ multiplied by the number of discretization nodes, where $d=dim\left(Y\right)$ is the reproduction dimension. Since achieving $\epsilon$-accuracy in d-dimensional integration necessitates a grid with $O(1/{\epsilon }^d)$ nodes, the overall complexity becomes $O\left({\displaystyle \frac{\left|\log \epsilon \right|}{{\epsilon }^{2d+1}}}(1+\log \left|\log \epsilon \right|)\right)$. Clearly, this leads to substantial computational complexity.

To mitigate the aforementioned substantial computational complexity, we design fast algorithms by exploiting the structure of uniform discretization. Specifically, for squared-error distortion, we leverage the Fast Fourier Transform (FFT) to accelerate the iterated convolutions in the CBA updates, achieving an acceleration factor of $\left|\log \epsilon \right|/{\epsilon }^d$. For absolute-error distortion, we devise a fast algorithm based on dynamic programming ideas, inspired by [20], which achieves an acceleration factor of $1/{\epsilon }^d$.

The remaining part of this article is organized as follows. Section 2 develops the continuous RD framework, formulates its discretizations, and establishes the convergence of the associated discrete schemes. Section 3 reviews the BA and CBA algorithms, and provides estimates on their required amount of arithmetic operations for attaining $\epsilon$-accuracy within the solution of the continuous RD problem. Section 4 further develops acceleration techniques, including FFT-based acceleration for squared-error distortion and nearly linear time acceleration for absolute-error distortion, with reduced time and memory costs. Section 5 concludes this article.

2. Discretization of Continuous RD Problems and Convergence Analysis

In this section, we address the discretization of continuous RD problems and analyze the convergence of the resulting discrete schemes.

2.1. Continuous RD Problems and Their Discretizations

Consider a continuous memoryless source $X\in \mathcal{X}$ with reproduction $Y\in \mathcal{Y}$, where $\mathcal{X},\mathcal{Y}$ are finite-dimensional Euclidean spaces. Denote the dimension of $\mathcal{Y}$ as d. Let p and $\mathit{r}$ denote the probability distributions of X and Y, respectively, and let $\rho :\mathcal{X}\times \mathcal{Y}\to {\mathbb{R}}_{\ge 0}$ be a distortion measure. For an average distortion threshold $D\ge 0$, the RD function [6,21] is

$$R\left(D\right)\triangleq \underset{P_{Y|X}}{\min }I(X;Y)\mathrm{s}.\mathrm{t}.{\mathbb{E}}_{P_{XY}}\left[\rho (X,Y)\right]\le D,$$

(1)

where $P_{XY}=P_X{P}_{Y|X}$. Define $F(\mathit{r},\beta )$ as

$$F(\mathit{r},\beta )\triangleq -\int \log \left[\int \exp \left(-\beta \rho (x,y)\right)\mathrm{d}\mathit{r}\left(y\right)\right]\mathrm{d}p\left(x\right)-\beta D.$$

(2)

A convenient max–min formulation of the RD problem (cf. [22] (Equation (7))) is

$$\underset{\beta \ge 0}{\max } \underset{\mathit{r}\in \mathcal{W}}{\min }F(\mathit{r},\beta ),$$

(3)

where $\mathcal{W}=\{\mathit{r}:\int \mathrm{d}\mathit{r}(y)=1\}$ and D denotes the average distortion threshold. Here $\mathit{r}$ is understood as a probability measure on $\mathcal{Y}$ equipped with the topology of weak convergence [23], for which it is well-known that for continuous sources under the squared-error distortion measure, the optimal reproduction distribution $\mathit{r}$ is usually discrete [17,24].

In (3), by fixing the multiplier $\beta$ and defining $f\left(\mathit{r}\right)$ as

$$f\left(\mathit{r}\right)\triangleq -\int \log \left[\int \exp \left(-\beta \rho (x,y)\right)\mathrm{d}\mathit{r}\left(y\right)\right]\mathrm{d}p\left(x\right),$$

(4)

the inner minimization of (3) can be rewritten as (see [17] (Equation (3)):

$$\underset{\mathit{r}\in \mathcal{W}}{\min }f\left(\mathit{r}\right).$$

(5)

Since $\mathit{r}\left(y\right)$ is defined over the continuum $\mathcal{W}$, it is common to discretize it on a uniform grid of points ${\left\{y_j^n\right\}}_{j=1}^n$ within ${[-n^{{\displaystyle \frac{1}{2d}}},n^{{\displaystyle \frac{1}{2d}}}]}^d$. We denote the discretization step size as h, i.e., $h=2n^{-{\displaystyle \frac{1}{2d}}}$, and the interval $[y_j^n-h/2,y_j^n+h/2]$ containing $y_j$ as $I_j$, noting that these intervals are disjoint. This thus leads to the following discrete form:

$$\underset{\mathit{r}:{\sum }_{j=1}^n{r}_j=1}{\min }-\int \log \left[\sum _{j=1}^n\exp (-\beta \rho (x,y_j^n))r_j\right]\mathrm{d}p\left(x\right).$$

(6)

Correspondingly, (3) corresponds to the following discrete form:

$$\underset{\beta \ge 0}{\max }\underset{\mathit{r}:{\sum }_{j=1}^n{r}_j=1}{\min }-\int \log \left[\sum _{j=1}^n\exp (-\beta \rho (x,y_j^n))r_j\right]\mathrm{d}p\left(x\right)-\beta D.$$

(7)

The solution of (6) or (7) can be represented as ${\mathit{r}}^n={\sum }_j{r}_j{\delta }_{y_j^n}$, where ${\delta }_{y_j^n}$ is the Dirac delta function at $y_j^n$.

Before proceeding with the subsequent discussion, we clarify some basic notations and terms used throughout the text. ${[-M,M]}^d$ denotes the d-fold Cartesian product of $[-M,M]$; $\parallel ·\parallel$ denotes the ${\ell }_{\infty }$ norm; $a_n\to a$ means convergence in $\mathbb{R}$, while $f_n\rightrightarrows f$ denotes uniform convergence [16], i.e., $\forall \epsilon >0$, $|f_n\left(x\right)-f\left(x\right)|\le \epsilon ,\forall x$ when n is sufficiently large. For the sequence ${\left\{{\mathit{r}}^n\right\}}_{n=1}^{\infty }$, the limit points [23] are all the points satisfying that every punctured neighborhood contains at least one point in ${\left\{{\mathit{r}}^n\right\}}_{n=1}^{\infty }$, and we denote the set that contains all the limit points of the sequence ${\left\{{\mathit{r}}^n\right\}}_{n=1}^{\infty }$ as $L\left({\left\{{\mathit{r}}^n\right\}}_{n=1}^{\infty }\right)$. We also write weak convergence as ${\mathit{r}}^k\stackrel{w}{\to }\mathit{r}$. For brevity, the expression $\log \int \exp (-\beta \rho (x,y\left)\right)\mathrm{d}\mathit{r}\left(y\right)$ will be abbreviated as ${\mathcal{H}}_{\mathit{r}\left(y\right)}\left(x\right)$ when no confusion arises. We write $\mathcal{X}={\left\{x_i\right\}}_{i=1}^m$ and $\mathcal{Y}={\left\{y_j\right\}}_{j=1}^n$ for discretized spaces $\mathcal{X}$ and $\mathcal{Y}$, respectively.

2.2. Convergence of Discrete Schemes

Before proceeding with the analysis, we outline the underlying assumption.

Assumption 1.

The distortion measure ρ and source distribution p satisfy

$$\begin{array}{cc}\hfill & \forall \beta >0,\forall \epsilon >0,\exists \delta >0, \mathit{s}.\mathit{t}.\forall x,\forall \parallel y_1-y_2\parallel <\delta ,\hfill \\ \hfill & |\exp (-\beta \rho (x,y_1))-\exp (-\beta \rho (x,y_2))|<\epsilon ,\int \underset{\parallel y\parallel \le A}{\max }\rho (x,y)\mathrm{d}p\left(x\right)<\infty ,\forall A>0,\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill & |\exp (-\beta \rho (x,y_1))\rho (x,y_1)-\exp (-\beta \rho (x,y_2))\rho (x,y_2)|<\epsilon .\hfill \end{array}$$

We emphasize that this assumption constitutes general conditions applicable to broad scenarios. With this assumption, we summarize the main results of convergence and convergence rate in the following theorem:

Theorem 1.

The convergence and convergence rate results for discrete problem (7) are as follows:

(1) 

Under the assumption, the solutions $({\mathit{r}}^n,{\beta }^n)$ to the discrete problem (7) satisfy convergence of the objective values and convergence of the sequence, i.e.,

$$F({\mathit{r}}^n,{\beta }^n)\to F^{*},andL\left({\left\{({\mathit{r}}^n,{\beta }^n)\right\}}_{n=1}^{\infty }\right)aresolutionsof\left(3\right),$$

where $F^{*}$ is the optimal value of the continuous problem (3).

(2) 

When the probability distribution p of X is supported over ${[-M,M]}^d$, the optimal values $F({\mathit{r}}^n,{\beta }^n)$ of the discrete problem (7) satisfy the error estimate

$$|F({\mathit{r}}^n,{\beta }^n)-F^{*}|\le Ch,$$

where $C=C(p,\rho )$ is a constant independent of n, and $h=2M/n^{{\displaystyle \frac{1}{d}}}$ is the discretization step size.

To prove part (1) in Theorem 1, we first consider the case of fixed $\beta$, establishing the convergence of the discrete problem (6) to the continuous problem (5).

Lemma 1.

Under the assumption, for any fixed multiplier $\beta >0$, the solutions ${\mathit{r}}^n$ to the discrete problem (6) satisfy both value convergence and sequence convergence to (5), i.e.,

$$f\left({\mathit{r}}^n\right)\to f^{*},andL\left({\left\{{\mathit{r}}^n\right\}}_{n=1}^{\infty }\right)isthesolutionsetof\left(5\right),$$

where $f^{*}$ is the optimal value of the continuous problem (5).

Proof. 

Detailed derivations are presented in Appendix A.1.    □

Next, we provide the convergence proof of the RD problem (3) via its discrete problem (7). Let the optimal solution of the continuous problem (3) and the discrete problem (7) be $({\mathit{r}}^{*},{\beta }^{*})\mathrm{and}({\mathit{r}}^n,{\beta }^n)$ respectively. Since ${\mathit{r}}^n$ has no concrete form, it is not straightforward to prove its convergence. The key idea is a transformation from the convergence of ${\mathit{r}}^n$ to that of a variable ${\mathit{q}}^n$ of certain concrete form, as summarized in Proposition 1 below, which has drawn some inspiration from [25].

Proposition 1.

Construct ${\mathit{q}}^n={\sum }_{j=1}^n{A}_j^n{\delta }_{y_j^n}$, where $A_j^n={\int }_{I_j}\mathrm{d}{\mathit{r}}^{*}$, and $I_j$ is defined in Section 2.1. To prove the convergence of the objective value in problem (3), i.e., $F({\mathit{r}}^n,{\beta }^n)\to F({\mathit{r}}^{*},{\beta }^{*})$, we only need to show $F({\mathit{q}}^n,{\tilde{\beta }}^n)\to F({\mathit{r}}^{*},{\beta }^{*})$, where ${\tilde{\beta }}^n={argmax}_{\beta }F({\mathit{q}}^n,\beta )$.

Proof. 

Detailed derivations are presented in Appendix A.2.    □

In the following Lemma 2, we provide the convergence result of ${\tilde{\beta }}^n$, as a prelude to the convergence proof for the RD problem.

Lemma 2.

Convergence properties of ${\tilde{\beta }}^n$:

(1) 

${\tilde{\beta }}^n$ is lower bounded away from zero for any n, i.e., $\exists B_1>0$, ${\tilde{\beta }}^n\ge B_1,\forall n$.

(2) 

${\tilde{\beta }}^n$ converges to the optimal solution ${\beta }^{*}$.

Proof 

(1). By using subsequence analysis [23] and uniform convergence analysis [16], we can obtain the result. A detailed derivation is presented in Appendix A.3.

(2). By using the sequentially compact property [23], the truncation technique and the tools of upper limits and lower limits [16], the convergence can be proven. Detailed derivations are presented in Appendix A.3.    □

At this point, we are ready to establish the convergence of the discrete problem (7) in Theorem 1.

Proof of Theorem 1(1).

We now show that $F({\mathit{q}}^n,{\tilde{\beta }}^n)\to F({\mathit{r}}^{*},{\beta }^{*})$. By Lemma 1, $F({\mathit{q}}^n,{\beta }^{*})\to F({\mathit{r}}^{*},{\beta }^{*})$. Therefore, it suffices to prove that

$$F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{q}}^n,{\beta }^{*})⟶0.$$

For this, we notice that

$$\begin{array}{cc}\hfill & 0\le F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{q}}^n,{\beta }^{*})=-\int \log {\displaystyle \frac{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}{\int \exp (-{\beta }^{*}\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}\mathrm{d}p\left(x\right)-({\tilde{\beta }}^n-{\beta }^{*})D.\hfill \end{array}$$

We divide the integral into two parts and estimate each of them. For simplicity, denote $H_{\mathit{r}}^{\beta }\left(x\right)=\int \exp (-\beta \rho (x,y))\mathrm{d}\mathit{r}\left(y\right)$.

$$\begin{array}{cc}\hfill & -{\int }_{\parallel x\parallel \le M}\log {\displaystyle \frac{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}{\int \exp (-{\beta }^{*}\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}\mathrm{d}p\left(x\right)\hfill \\ \hfill & ={\int }_{\parallel x\parallel \le M}\log \left({\displaystyle \frac{H_{{\mathit{q}}^n}^{{\beta }^{*}}\left(x\right)-H_{{\mathit{q}}^n}^{{\tilde{\beta }}^n}\left(x\right)}{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}+1\right)\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{|H_{{\mathit{q}}^n}^{{\beta }^{*}}\left(x\right)-H_{{\mathit{q}}^n}^{{\tilde{\beta }}^n}\left(x\right)|}{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}\mathrm{d}p\left(x\right)\le {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{|H_{{\mathit{q}}^n}^{{\beta }^{*}}\left(x\right)-H_{{\mathit{q}}^n}^{{\tilde{\beta }}^n}\left(x\right)|}{{\delta }_0}}\mathrm{d}p\left(x\right),\hfill \end{array}$$

(8)

where ${\delta }_0$ is a lower bound and

$$\begin{array}{cc}\hfill \int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)& \ge {\int }_{\parallel y\parallel \le M}\exp (-B_0\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\hfill \\ \hfill & \ge {\int }_{\parallel y\parallel \le M}\exp (-B_0{\rho }^{*})\mathrm{d}{\mathit{q}}^n\left(y\right)\ge {\displaystyle \frac{1}{2}}\exp (-B_0{\rho }^{*})\triangleq {\delta }_0.\hfill \end{array}$$

Here, $B_0$ is an upper bound of ${\tilde{\beta }}^n$, since it is convergent, M is larger than the constant A in Lemma 1, and ${\rho }^{*}$ is a constant equal to ${\max }_{x,y\in {[-M,M]}^d}\rho (x,y)$. Using the same derivation as (A7) in the proof of Lemma 2, we have

$$\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)-\int \exp (-{\beta }^{*}\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\rightrightarrows 0.$$

Thus, when n is sufficiently large,

$${\int }_{\parallel x\parallel \le M}{\displaystyle \frac{|H_{{\mathit{q}}^n}^{{\beta }^{*}}\left(x\right)-H_{{\mathit{q}}^n}^{{\tilde{\beta }}^n}\left(x\right)|}{{\delta }_0}}\mathrm{d}p\left(x\right)\le \epsilon .$$

We obtain

$$\underset{n}{\lim }-{\int }_{\parallel x\parallel \le M}\log {\displaystyle \frac{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}{\int \exp (-{\beta }^{*}\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}\mathrm{d}p\left(x\right)=0.$$

Meanwhile, let $g\left(x\right)={\max }_{\parallel y\parallel \le A}\rho (x,y)$, then

$$\begin{array}{cc}\hfill -{\int }_{\parallel x\parallel >M}\log {\displaystyle \frac{H_{{\mathit{q}}^n}^{{\tilde{\beta }}^n}\left(x\right)}{H_{{\mathit{q}}^n}^{{\beta }^{*}}\left(x\right)}}\mathrm{d}p\left(x\right)& ={\int }_{\parallel x\parallel >M}\left[-\log \left(H_{{\mathit{q}}^n}^{{\tilde{\beta }}^n}\left(x\right)\right)-\log \left(H_{{\mathit{q}}^n}^{{\beta }^{*}}\left(x\right)\right)\right]\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel >M}\left[-\log \left({\displaystyle \frac{1}{2}}{e}^{-{\tilde{\beta }}^{n}g\left(x\right)}\right)-\log \left({\displaystyle \frac{1}{2}}{e}^{-{\beta }^{*}g\left(x\right)}\right)\right]\mathrm{d}p\left(x\right)\hfill \\ \hfill & ={\int }_{\parallel x\parallel >M}\left[2\log 2+{\tilde{\beta }}^{n}g\left(x\right)+{\beta }^{*}g\left(x\right)\right]\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel >M}\left[2\log 2+B_{0}g\left(x\right)+{\beta }^{*}g\left(x\right)\right]\mathrm{d}p\left(x\right).\hfill \end{array}$$

Here, we used the fact

$$\begin{array}{cc}\hfill & H_{{\mathit{q}}^n}^{{\tilde{\beta }}^n}\left(x\right)\ge {\int }_{\parallel y\parallel \le A}\exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\hfill \\ \hfill & \ge {\int }_{\parallel y\parallel \le A}\exp (-{\tilde{\beta }}^{n}g\left(x\right))\mathrm{d}{\mathit{q}}^n\left(y\right)\ge 1/2\exp (-{\tilde{\beta }}^{n}g\left(x\right)).\hfill \end{array}$$

Combining the two parts and taking the limit $n\to \infty$, we obtain

$$\begin{array}{c}\underset{n}{\lim \sup }[F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{q}}^n,{\beta }^{*})] \hfill \\ =\underset{n}{\lim \sup }-\int \log {\displaystyle \frac{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}{\int \exp (-{\beta }^{*}\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}\mathrm{d}p\left(x\right)\hfill \\ =\underset{n}{\lim \sup }\left(-{\int }_{\parallel y\parallel >M}\log {\displaystyle \frac{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}{\int \exp (-{\beta }^{*}\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}\mathrm{d}p\left(x\right)\right)\hfill \\ \le {\int }_{\parallel x\parallel >M}\left[2\log 2+B_{0}g\left(x\right)+{\beta }^{*}g\left(x\right)\right]\mathrm{d}p\left(x\right).\hfill \end{array}$$

Then by taking the limit of $M\to \infty$, we obtain the convergence. Here we used the assumption, i.e., $\int g\left(x\right)\mathrm{d}p\left(x\right)<\infty$.

To prove the convergence of the solutions, let $(\tilde{\mathit{r}},\tilde{\beta })$ be a limit point of the solution sequence ${\left\{({\mathit{r}}^n,{\beta }^n)\right\}}_{n=1}^{\infty }$. Then there exists a subsequence ${\left\{({\mathit{r}}^{n_k},{\beta }^{n_k})\right\}}_{k=1}^{\infty }$ satisfying ${\mathit{r}}^{n_k}\stackrel{w}{\to }\tilde{\mathit{r}}$ and ${\beta }^{n_k}\to \tilde{\beta }$. Next, we have

$$F(\tilde{\mathit{r}},\tilde{\beta })=\underset{k}{\lim }F({\mathit{r}}^{n_k},{\beta }^{n_k})=F({\mathit{r}}^{*},{\beta }^{*}),$$

since we have proven that ${\lim }_{n}F({\mathit{r}}^n,{\beta }^n)=F({\mathit{r}}^{*},{\beta }^{*})$. By the optimality property of $({\mathit{r}}^{*},{\beta }^{*})$ and $({\mathit{r}}^{n_k},{\beta }^{n_k})$,

$$F(\tilde{\mathit{r}},\tilde{\beta })=F({\mathit{r}}^{*},{\beta }^{*})\le F({\mathit{r}}^{n_k},{\beta }^{*})\le F({\mathit{r}}^{n_k},{\beta }^{n_k})\le F({\overline{\mathit{q}}}^{n_k},{\beta }^{n_k}).$$

Here, ${\overline{\mathit{q}}}^{n_k}\in W_{n_k}$ is a discrete version of $\overline{\mathit{r}}={\mathrm{argmin}}_{\mathit{r}\in W}F(\mathit{r},\tilde{\beta })$, i.e., ${\overline{\mathit{q}}}^{n_k}={\sum }_j{\int }_{I_j}d\overline{\mathit{r}}{\delta }_{y_j^{n_k}}$. Similarly to the preceding analysis, we have $F({\overline{\mathit{q}}}^{n_k},{\beta }^{n_k})\to F(\overline{\mathit{r}},\tilde{\beta })$. Thus,

$$F(\tilde{\mathit{r}},\tilde{\beta })\le F(\overline{\mathit{r}},\tilde{\beta })\le F(\mathit{r},\tilde{\beta }),\forall \mathit{r}\in W.$$

This means $\tilde{\mathit{r}}={\mathrm{argmin}}_{\mathit{r}\in W}F(\mathit{r},\tilde{\beta })$ and

$$F(\tilde{\mathit{r}},\tilde{\beta })=\underset{\mathit{r}\in W}{\min }F(\mathit{r},\tilde{\beta })\triangleq h\left(\tilde{\beta }\right).$$

However, $F(\tilde{\mathit{r}},\tilde{\beta })=F({\mathit{r}}^{*},{\beta }^{*})=h\left({\beta }^{*}\right)$ and $h\left({\beta }^{*}\right)={\max }_{\beta \ge 0}h\left(\beta \right)$ hold due to the optimality property of $({\mathit{r}}^{*},{\beta }^{*})$. So, we obtain that $\tilde{\beta }$ is optimal, i.e., $\tilde{\beta }={\mathrm{argmax}}_{\beta \ge 0}h\left(\beta \right)$. Further combining $\tilde{\mathit{r}}={\mathrm{argmin}}_{r\in W}F(\mathit{r},\tilde{\beta })$, we obtain that $(\tilde{\mathit{r}},\tilde{\beta })$ is optimal.    □

To prove part (2) in Theorem 1, we first establish the convergence rate for the discrete problems (6).

Lemma 3.

When the probability distribution p of X is supported over ${[-M,M]}^d$, the optimal values $f\left({\mathit{r}}^n\right)$ of the discrete problem (6) satisfy the error estimate

$$|f\left({\mathit{r}}^n\right)-f^{*}| \le Ch,$$

where C is a constant and $h=2M/n^{{\displaystyle \frac{1}{d}}}$ is the discretization step size.

Proof. 

By refining the estimation in Lemma 1, we obtain the convergence rate. A derivation is shown in Appendix A.4.    □

Correspondingly, the convergence rate of problem (7) can be proved as follows.

Proof of Theorem 1(2).

Since $\mathit{r}$ is supported over ${[-M,M]}^d$, we can simply take the equidistant discretization nodes ${\left\{y_j^n\right\}}_{j=1}^n$ from ${[-M,M]}^d$. Due to the inequality (A5) in the proof of Proposition 1,

$$0\le F({\mathit{r}}^n,{\beta }^n)-F({\mathit{r}}^{*},{\beta }^{*})\le F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{r}}^{*},{\beta }^{*}).$$

Thus, we only need to evaluate $F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{r}}^{*},{\beta }^{*})$. Assume for $x,y\in {[-M,M]}^d$ that $e^{-\beta \rho (x,y)}$ and $e^{-\beta \rho (x,y)}\rho (x,y)$ are Lipschitz continuous with the common Lipschitz constant L, for $0<B_1\le \beta \le B_0$.

First, we estimate the convergence rate of ${\tilde{\beta }}^n\to {\beta }^{*}$:

$$\begin{array}{cc}\hfill & |\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)-\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)|\hfill \\ \hfill & \le \sum _i{\int }_{I_i}|\exp (-{\tilde{\beta }}^n\rho (x,y))-\exp (-{\tilde{\beta }}^n\rho (x,y_i^n))|\mathrm{d}{\mathit{r}}^{*}\left(y\right)\hfill \\ \hfill & \le \sum _i{\int }_{I_i}L\parallel y-y_i^n\parallel \mathrm{d}{\mathit{r}}^{*}\left(y\right)\le Lh/2.\hfill \end{array}$$

Here, we used the fact ${\bigcup }_{i=1}^n{I}_i\supseteq {[-M,M]}^d$, and the bound on ${\tilde{\beta }}^n$, i.e., $0<B_1\le {\tilde{\beta }}^n\le B_0$.

Similarly,

$$|\int \exp (-{\tilde{\beta }}^n\rho (x,y))\rho (x,y)\mathrm{d}{\mathit{q}}^n\left(y\right)-\int \exp (-{\tilde{\beta }}^n\rho (x,y))\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)|\le Lh/2.$$

Next, as in the proof of Theorem 1(1), we obtain the lower bounds

$$\int e^{-{\tilde{\beta }}^n\rho (x,y)}\mathrm{d}{\mathit{q}}^n\left(y\right)\ge {\delta }_0,\int e^{-{\tilde{\beta }}^n\rho (x,y)}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\ge {\delta }_0.$$

Moreover, since ${\tilde{\beta }}^n\ge B_1>0$ and $e^{-\beta \rho }$ is decreasing in $\beta$,

$$\begin{array}{cc}\hfill \int e^{-{\tilde{\beta }}^n\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)& \le \int e^{-B_1\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)\le \int M_1\mathrm{d}{\mathit{r}}^{*}\left(y\right)=M_1,\hfill \end{array}$$

where $M_1$ is the upper bound on the function $\exp (-B_{1}t)t,t\ge 0$.

Now, since $G_{{\mathit{r}}^{*}}\left({\beta }^{*}\right)=0=G_{{\mathit{q}}^n}\left({\tilde{\beta }}^n\right)$, we have the following estimate:

$$\begin{array}{cc}\hfill |G_{{\mathit{r}}^{*}}\left({\tilde{\beta }}^n\right)-G_{{\mathit{r}}^{*}}\left({\beta }^{*}\right)|& =|G_{{\mathit{r}}^{*}}\left({\tilde{\beta }}^n\right)-G_{{\mathit{q}}^n}\left({\tilde{\beta }}^n\right)|\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}|{\displaystyle \frac{\left(\int \exp \left(-{\tilde{\beta }}^n\rho (x,y)\right)\rho (x,y)\mathrm{d}{q}^n\left(y\right)\right)}{\left(\int \exp \left(-{\tilde{\beta }}^n\rho (x,y)\right)\mathrm{d}{q}^n\left(y\right)\right)}}\hfill \\ \hfill & -{\displaystyle \frac{\left(\int \exp \left(-{\tilde{\beta }}^n\rho (x,y)\right)\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}{\left(\int \exp \left(-{\tilde{\beta }}^n\rho (x,y)\right)\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}}|\mathrm{d}p\left(x\right)\hfill \\ \hfill & ={\int }_{\parallel x\parallel \le M}|{\displaystyle \frac{b_1}{a_1}}-{\displaystyle \frac{b_2}{a_2}}|\mathrm{d}p\left(x\right)={\int }_{\parallel x\parallel \le M}{\displaystyle \frac{|a_2{b}_1-a_1{b}_2|}{a_1{a}_2}}\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{a_2|b_1-b_2|+b_2|a_2-a_1|}{a_1{a}_2}}\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}\left({\displaystyle \frac{Lh}{2{\delta }_0}}+{\displaystyle \frac{M_{1}Lh/2}{{\delta }_0^2}}\right)\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\displaystyle \frac{Lh}{2{\delta }_0}}+{\displaystyle \frac{M_{1}Lh/2}{{\delta }_0^2}}=O\left(h\right),\hfill \end{array}$$

wherein for the simplicity of notation, we write $a_1,a_2,b_1,b_2$ for the respective integrals.

Next, we provide the estimate of ${\tilde{\beta }}^n-{\beta }^{*}$. Let $-L_1=G_{{\mathit{r}}^{*}}^{\prime }\left({\beta }^{*}\right)<0$. Then there exists ${\delta }_2>0$ such that, for all $\beta \in ({\beta }^{*}-{\delta }_2,{\beta }^{*}+{\delta }_2)$,

$$G_{{\mathit{r}}^{*}}^{\prime }\left(\beta \right)\le -{\displaystyle \frac{L_1}{2}},$$

and ${\tilde{\beta }}^n$ lies in this neighborhood once n is sufficiently large. Therefore, by the Lagrange mean value theorem,

$$|G_{{\mathit{r}}^{*}}\left({\tilde{\beta }}^n\right)-G_{{\mathit{r}}^{*}}\left({\beta }^{*}\right)|=\left|G_{{\mathit{r}}^{*}}^{\prime }\left(\zeta \right)\right||{\tilde{\beta }}^n-{\beta }^{*}|\ge {\displaystyle \frac{L_1}{2}}|{\tilde{\beta }}^n-{\beta }^{*}|.$$

Here, $\zeta$ is a real number between ${\tilde{\beta }}^n$ and ${\beta }^{*}$. Thus,

$$|{\tilde{\beta }}^n-{\beta }^{*}|\le {\displaystyle \frac{2}{L_1}}|G_{{\mathit{r}}^{*}}\left({\tilde{\beta }}^n\right)-G_{{\mathit{r}}^{*}}\left({\beta }^{*}\right)|=O\left(h\right).$$

Now we provide the convergence rate of $F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{r}}^{*},{\beta }^{*})$:

$$|F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{r}}^{*},{\beta }^{*})|\le |F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{q}}^n,{\beta }^{*})|+|F({\mathit{q}}^n,{\beta }^{*})-F({\mathit{r}}^{*},{\beta }^{*})|.$$

By conducting the following evaluation

$$\begin{array}{cc}\hfill & |\int \exp (-{\beta }^{*}\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)-\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)|\hfill \\ \hfill & \le \int e^{-{\beta }^{*}\rho (x,y)}|e^{({\beta }^{*}-{\tilde{\beta }}^n)\rho (x,y)}-1|\mathrm{d}{q}^n\left(y\right)\hfill \\ \hfill & \le {\int }_{\parallel y\parallel \le M}(e^{|{\beta }^{*}-{\tilde{\beta }}^n|\rho (x,y)}-1)\mathrm{d}{q}^n\left(y\right)\hfill \\ \hfill & \le {\int }_{\parallel y\parallel \le M}(e^{|{\beta }^{*}-{\tilde{\beta }}^n|{\rho }^{*}}-1)\mathrm{d}{q}^n\left(y\right)\le (e^{|{\beta }^{*}-{\tilde{\beta }}^n|{\rho }^{*}}-1)\hfill \\ \hfill & \le 2|{\beta }^{*}-{\tilde{\beta }}^n|{\rho }^{*}=O\left(h\right),\mathrm{when}n\mathrm{is}\mathrm{sufficiently}\mathrm{large},\hfill \end{array}$$

where ${\rho }^{*}={\max }_{x,y\in {[-M,M]}^d}\rho (x,y)$, we thus have the following estimate using the derivation in (8):

$$\begin{array}{cc}\hfill & 0\le F({\mathit{q}}^n,{\tilde{\beta }}^n)-F({\mathit{q}}^n,{\beta }^{*})\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{|H_{{\mathit{q}}^n}^{{\beta }^{*}}\left(x\right)-H_{{\mathit{q}}^n}^{{\tilde{\beta }}^n}\left(x\right)|}{{\delta }_0}}\mathrm{d}p\left(x\right)-({\tilde{\beta }}^n-{\beta }^{*})D\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}O\left(h\right)/{\delta }_0\mathrm{d}p\left(x\right)+O\left(h\right)=O\left(h\right),\hfill \end{array}$$

where we denote $H_{\mathit{r}}^{\beta }\left(x\right)=\int \exp (-\beta \rho (x,y))\mathrm{d}\mathit{r}\left(y\right)$. Meanwhile, due to Lemma 3, $|F({\mathit{q}}^n,{\beta }^{*})-F({\mathit{r}}^{*},{\beta }^{*})|=O(h)$. Thus, we obtain the convergence rate $\left|F\right({\mathit{r}}^n,{\beta }^n)-F({\mathit{r}}^{*},{\beta }^{*})|=O\left(h\right)=O(1/n^{{\displaystyle \frac{1}{d}}}).$    □

Remark 1.

We first want to clarify that Theorem 1, as presented here, is specifically centered on the CBA algorithm. Additionally, it should be noted that corresponding results for the BA algorithm are also covered: both Lemma 1 and Lemma 3 entail the same conclusions for the BA algorithm.

2.3. Numerical Verfication

We conduct experiments on a uniform source to confirm the convergence shown in this section. We consider the uniform source on interval $[-8,8]$ and solve the corresponding discrete problems with different discretization parameters, with the node number varying according to $n=20,40,80,160$ for the discretization of Y. In order to ensure accuracy for evaluating the integrals with respect to $p\left(x\right)$, we fix a sufficiently large node number $m=300$ for X. We use both the BA and CBA algorithms to solve the discrete problems (6) and (7), respectively. The corresponding results of the reproduction distribution are illustrated in Figure 1.

Figure 1. The discrete optimal reproduction distribution produced by the BA algorithm for slope $\beta =0.1$ (upper left) and $\beta =0.2$ (upper right), and by the CBA algorithm for target distortion $D=4$ (lower left) and $D=3$ (lower right). For clearer visualization, the ${\log }_2$ scale of the density is used on the vertical axis.

It has been well known (see, i.e., [17]) that the optimal reproduction distribution of the uniform source is discrete. From Figure 1, the convergence of the solutions of discrete problems to discrete reproduction distributions is clearly demonstrated, as the discretization becomes finer progressively.

3. Discrete RD Algorithms and Their $\mathit{\epsilon }$-Accuracy Analysis

In the previous section, we have shown that for computing the RD function of a continuous source, as discretization step size asymptotically vanishes, solutions to discrete problems converge to those of the continuous problem. Discrete problems are typically solved via BA [10,11] and CBA [12] algorithms. In this section, we review these algorithms, and provide estimates on their required amount of arithmetic operations for attaining $\epsilon$-accuracy within the solution of the continuous problem.

3.1. BA Algorithm

As a classical numerical method for computing the RD function, the BA algorithm [10,11] minimizes the RD Lagrangian with a fixed penalty parameter. Concretely, for a fixed penalty $\beta >0$, it minimizes

$${\mathcal{L}}_{\mathrm{RD}}^{\beta }\left(P_{Y|X}\right)\triangleq I(X;Y)+\beta {\mathbb{E}}_{P_{XY}}\left[d(X,Y)\right].$$

Geometrically, $\beta$ is the slope of a tangent line to the RD curve $R\left(D\right)$; hence, minimizing ${\mathcal{L}}_{\mathrm{RD}}^{\beta }$ at a fixed $\beta$ yields a conditional distribution $P_{Y|X}^{*}$ associated with the point of tangency $(D_{\lambda },R_{\lambda })$ on the RD curve [26].

Write $w\left(y\right|x)$ for the conditional probability $P_{Y|X}$, and $\mathit{r}\left(y\right)$ for the marginal distribution of Y. The BA algorithm alternates between the two variables $w\left(y\right|x)$ and $\mathit{r}\left(y\right)$ in the following way [10]:

$$\begin{array}{cc}\hfill w\left(y\right|x)& =\left(\mathit{r}\left(y\right)e^{-\beta \rho (x,y)}\right)/\int e^{-\beta \rho (x,y)}\mathit{r}\left(y\right)\mathrm{d}y,\hfill \\ \hfill \mathit{r}\left(y\right)& =\int p(x)w(y|x)\mathrm{d}x.\hfill \end{array}$$

After discretization, the alternating iterations can be reformulated as:

$$\begin{array}{cc}\hfill w_{ij}& =\left(r_j{e}^{-\beta {\rho }_{ij}}\right)/\sum _{j=1}^n{e}^{-\beta {\rho }_{ij}}{r}_j,i=1,2\cdots m,j=1,2\cdots n,\hfill \end{array}$$

(9a)

$$\begin{array}{cc}\hfill r_j& =\sum _{i=1}^m{p}_i{w}_{ij},j=1,2\cdots n,\hfill \end{array}$$

(9b)

where $w_{ij}=w\left(y_j\right|x_i)h,p_i=p\left(x_i\right)h,r_j=\mathit{r}\left(y_j\right)h,{\rho }_{ij}=\rho (x_i,y_j)$, and h is the discretization step size of uniform discretization.

It has been shown in the literature (see, e.g., [26] (Chapter 8)) that, for discrete sources, the iterative procedure of the BA algorithm converges to the RD function at rate $O(1/n)$. Furthermore, as shown in [13] (Theorem 2, Equation (96)), the BA algorithm needs $O(\log n/\epsilon )$ iterations to achieve $\epsilon$-accuracy. In each iteration, the BA algorithm performs two matrix–vector multiplications, involving $O\left(mn\right)$ arithmetic operations. Moreover, the space cost amounts to $O\left(mn\right)$ since we need to store $w_{ij}$ and $r_j,i,=1,2,\cdots ,m,j=1,2\cdots n$.

Based on the convergence rate analysis in Lemma 3, we conduct a complexity analysis to estimate the computational cost of solving the continuous problem (5) to achieve $\epsilon$-accuracy via the BA algorithm.

Theorem 2.

To ensure ε-accuracy for computing the continuous RD function, the BA algorithm needs $O\left({\displaystyle \frac{m\left|\log \epsilon \right|}{{\epsilon }^{d+1}}}\right)$ arithmetic operations. Here, m is the number of discretization nodes of X when conducting numerical integration and d is the dimension of the reproduction space $\mathcal{Y}$.

Proof. 

By Theorem 3, to ensure that $f\left({\mathit{r}}^n\right)-f\left({\mathit{r}}^{*}\right)\le \epsilon$, the sample size n must satisfy $1/n^{1/d}\sim \epsilon$, i.e., $n\sim {\epsilon }^{-d}$. Subsequently, the BA algorithm is applied to solve the associated discrete problem up to an error tolerance at level $\epsilon$. As shown in [13] (Theorem 2, Equation (96)), the BA algorithm needs $O(\log n/\epsilon )$ iterations to achieve $\epsilon$-accuracy. In each iteration, the BA algorithm alternates between $w\left(y_i^n\right|x)$ and $r\left(y_i^n\right)$ in the following way:

$$\begin{array}{cc}\hfill w\left(y_i^n\right|x)& =\left(r\left(y_i^n\right)e^{-\beta \rho (x,y_i^n)}\right)/\sum _{i=1}^n{e}^{-\beta \rho (x,y_i^n)}r\left(y_i^n\right),\hfill \\ \hfill r\left(y_i^n\right)& =\int p\left(x\right)w\left(y_i^n\right|x)\mathrm{d}x.\hfill \end{array}$$

Let $x_1,x_2\cdots x_m$ be the nodes of numerical integration with respect to x. When computing $w\left(y_i^n\right|x)$, we need to perform matrix–vector multiplication

$$\sum _{i=1}^n{e}^{-\beta \rho (x_j,y_i^n)}r\left(y_i^n\right),j=1,2\cdots m,$$

and it involves $O\left(mn\right)$ arithmetic operations. The total computation cost for $w\left(y_i^n\right|x)$ is $O\left(mn\right)$. When computing $r\left(y_i^n\right)$, we need to perform numerical integration,

$$\int p\left(x\right)w\left(y_i^n\right|x)\mathrm{d}x\sim \sum _{j=1}^m{A}_{j}p\left(x_j\right)w\left(y_i^n\right|x_j),i=1,2\cdots n,$$

where $A_j$ are the numerical integration coefficients. This is a matrix–vector multiplication, and it involves $O\left(mn\right)$ arithmetic operations. Thus, the computational cost for each iteration in the BA algorithm is $O\left(mn\right)$. Hence, the total computational cost is $O(mn\log n/\epsilon )=O\left(m\right|\log \epsilon |/{\epsilon }^{d+1})$.    □

3.2. CBA Algorithm

Inspired by the connection between OT and RD [27,28], Chen et al. [12] introduced the CBA algorithm [12], which explicitly incorporates the expected distortion constraint of the RD function. This allows the Lagrange multiplier $\beta$ to be updated via a one-dimensional root-finding step to meet a target distortion D in each iteration, thereby solving the original RD problem (1) directly, without sweeping the entire RD curve as in the BA algorithm. Specifically, when alternating between the two variables $w\left(y\right|x)$ and $\mathit{r}\left(y\right)$, the CBA algorithm updates the multiplier $\beta$ (see [12] for further details):

$$\begin{array}{cc}\hfill \mathrm{Solve}G\left(\beta \right)& \triangleq \int {\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathit{r}\left(y\right)\mathrm{d}y\right)}{\left(\int e^{-\beta \rho (x,y)}\mathit{r}\left(y\right)\mathrm{d}y\right)}}p\left(x\right)\mathrm{d}x-D=0,\hfill \\ \hfill w\left(y\right|x)& =\left(\mathit{r}\left(y\right)e^{-\beta \rho (x,y)}\right)/\int e^{-\beta \rho (x,y)}\mathit{r}\left(y\right)\mathrm{d}y,\hfill \\ \hfill \mathit{r}\left(y\right)& =\int p(x)w(y|x)\mathrm{d}x,\hfill \end{array}$$

where $\beta$ can be efficiently evaluated by Newton’s method since $G\left(\beta \right)$ is monotone, as shown in [12]. After discretization, the iterative procedure turns into:

$$\begin{array}{cc}\hfill \mathrm{Solve}G\left(\beta \right)& \triangleq \sum _{i=1}^m{\displaystyle \frac{\left({\sum }_{j=1}^n{e}^{-\beta {\rho }_{ij}}{\rho }_{ij}{r}_j\right)}{\left({\sum }_{j=1}^n{e}^{-\beta {\rho }_{ij}}{r}_j\right)}}{p}_i-D=0,\hfill \end{array}$$

(10a)

$$\begin{array}{cc}\hfill w_{ij}& =\left(r_j{e}^{-\beta {\rho }_{ij}}\right)/\sum _{j=1}^m{e}^{-\beta {\rho }_{ij}}{r}_j,i=1,2,\cdots ,m,j=1,2\cdots n,\hfill \end{array}$$

(10b)

$$\begin{array}{cc}\hfill r_j& =\sum _{i=1}^m{p}_i{w}_{ij},j=1,2\cdots n,\hfill \end{array}$$

(10c)

where $w_{ij}=w\left(y_j\right|x_i)h,p_i=p\left(x_i\right)h,r_j=r\left(y_j\right)h,{\rho }_{ij}=\rho (x_i,y_j)$ and h is the discretization step sizes. As shown in [12], for discrete sources, the iterative procedure of the CBA algorithm also converges to the RD function at rate $O(1/n)$. Note that differently from the BA algorithm which minimizes the RD Lagrangian for a given multiplier $\lambda$, the CBA algorithm directly reaches the RD function for a given target distortion D. By the analysis in [12], in each iteration, the CBA algorithm takes $O\left(mn(1+\log |\log \epsilon \left|\right)\right)$ arithmetic operations to achieve $\epsilon$-accuracy for solving $\beta$, while the space cost is also $O\left(mn\right)$.

Analogous to Theorem 2, we provide the complexity analysis to estimate the computational cost of solving the continuous problem (3) to achieve $\epsilon$-accuracy via the CBA algorithm.

Theorem 3.

To ensure ε-accuracy for computing the continuous RD function, the CBA algorithm needs $O\left({\displaystyle \frac{m\left|\log \epsilon \right|}{{\epsilon }^{d+1}}}(1+\log |\log \epsilon \left|\right)\right)$ arithmetic operations. Here, m is the number of discretization nodes of X when conducting numerical integration and d is the dimension of the reproduction space $\mathcal{Y}$.

Proof. 

By Lemma 3, to ensure that $F({\mathit{r}}^n,{\beta }^n)-F({\mathit{r}}^{*},{\beta }^{*})\le \epsilon$, the sample size n must satisfy $1/n^{1/d}\sim \epsilon$, i.e., $n\sim 1/{\epsilon }^d$. Then when using the CBA algorithm to solve the discrete problem within $\epsilon$ error tolerance, as shown in [12], we need $O\left({\displaystyle \frac{mn\log n}{\epsilon }}(1+\log |\log \epsilon \left|\right)\right)$ arithmetic operations to achieve $\epsilon$-accuracy. Thus, the computational cost is

$$O\left({\displaystyle \frac{mn\log n}{\epsilon }}(1+\log |\log \epsilon \left|\right)\right)=O\left({\displaystyle \frac{m\left|\log \epsilon \right|}{{\epsilon }^{d+1}}}(1+\log |\log \epsilon \left|\right)\right),$$

thereby completing the proof.    □

4. Acceleration Techniques for the CBA Algorithm

The previous section detailed the computational cost for discrete algorithms (i.e., BA and CBA) to obtain $\epsilon$-accurate RD solutions for continuous sources. To this end, we exploit the structure of specific distortion measures (i.e., squared-error and absolute-error) for developing fast algorithms. For clarity, we present the algorithms for the case of $m=n$; the case of $m\ne n$ follows with minor modifications.

4.1. Squared-Error Distortion

We begin with the one-dimensional case, i.e., $d=1$. Following the discretization form introduced in Section 3, we have for the squared-error distortion $\rho (x,y)={(x-y)}^2$:

$${\rho }_{ij}={(x_i-y_j)}^2=h^2{(i-j)}^2,K_{ij}\triangleq e^{-\beta {\rho }_{ij}}=e^{-\beta h^2{(i-j)}^2}.$$

For simplicity, we denote by $D_{\rho }$ the matrix whose $(i,j)$-entry is ${\rho }_{ij}$. Noting that $K_{ij}$ only depends on the difference between its indices, consequently, the two weighted sums in the CBA iterative procedure become linear convolutions and can be implemented in $O\left(m\log m\right)$ time by FFT with zero padding.

To begin, we clarify the notation and definitions. We refer to ∗ as linear convolution, and “$./$” and “⊙” denote element-wise division and multiplication, respectively. Here we denote the standard FFT operation and the standard inverse FFT operation as $FFT$ and $IFFT$, respectively.

For a given matrix K, $K_d$ denotes the even extension of its first column, constructed by concatenating the first column of K with the reversed first row of K, excluding the first entry. Thus, we can compute each convolution $K\ast v$ in $O\left(m\log m\right)$ time via the FFT [29,30]. Specifically, the vector v is zero-padded [31] to length $L=2m-1$ and $K_d$ is constructed; their discrete Fourier transforms are then computed, multiplied component-wise in the Fourier domain, and followed by the inverse transform. The leading m coefficients are retained to recover the linear convolution. This process is referred to as the function $\mathrm{FFT}\_\mathrm{conv}(K,v)$.

Substituting (10b) into (10c), the iterations of the CBA algorithm turn into:

$$\begin{array}{cc}\hfill \mathrm{Solve}G\left(\beta \right)& =\sum _{i=1}^m{p}_i{\displaystyle \frac{{\sum }_j{K}_{ij}{\rho }_{ij}{r}_j}{{\sum }_j{K}_{ij}{r}_j}}-D=0,\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill r_j& =r_j\sum _{i=1}^m{\displaystyle \frac{p_i{K}_{ij}}{{\sum }_{j=1}^m{K}_{ij}{r}_j}},j=1,2\cdots m.\hfill \end{array}$$

(11b)

We adopt the notation $K^{\left(0\right)}=K,K^{\left(2\right)}=K\odot D_{\rho },K^{\left(4\right)}=K\odot D_{\rho }\odot D_{\rho }$, and let $a=K^{\left(0\right)}\ast r$ and $u=K^{\left(2\right)}\ast r$. Then, $\beta$ can be updated by

$$\mathrm{Solve}G\left(\beta \right)=\sum _{i=1}^m{p}_i{\displaystyle \frac{u_i}{a_i}}-D=0.$$

(12)

Differentiating $G\left(\beta \right)$ yields

$$G^{\prime }\left(\beta \right)=\sum _{i=1}^m{p}_i{\displaystyle \frac{u_i^2-a_i{v}_i}{a_i^2}},v=K^{\left(4\right)}\ast r.$$

Note that $G^{\prime }\left(\beta \right)$ and $G\left(\beta \right)$ require three FFT-based convolutions $(a,u,v)$, which can be calculated within $O\left(m\log m\right)$ arithmetic operations. Thus, each Newton’s iteration involves $O\left(m\log m\right)$ arithmetic operations. Due to the locally quadratic convergence of Newton’s method, we need $\log \left|\log \epsilon \right|$ iterations to achieve $\epsilon$-accuracy. Hence, we only need $O\left(m\log \right|\log \epsilon \left|\right)$ arithmetic operations to implement (11a). Additionally, the update of (11b) requires two FFT-based convolutions,

$$a=K^{\left(0\right)}\ast r,b=p./a,s=K^{\left(0\right)}\ast b,r\leftarrow r\odot s,$$

(13)

which can also be calculated within $O\left(m\log m\right)$ arithmetic operations. Therefore, in each iteration of (11), it involves $O\left(m\log m(1+\log |\log \epsilon \left|\right)\right)$ arithmetic operations. Moreover, due to [12] (Theorem 3.2), we need $O(\log m/\epsilon )$ iterations. Therefore, the total computational cost of our method is $O\left({\displaystyle \frac{m{\left(\log m\right)}^2}{\epsilon }}(1+\log |\log \epsilon \left|\right)\right)$. Since $m\sim 1/\epsilon$ is needed to ensure the accuracy of numerical integration, the computational cost turns out to be $O\left({\displaystyle \frac{{\left|\log \epsilon \right|}^2}{{\epsilon }^2}}(1+\log |\log \epsilon \left|\right)\right)$ to achieve $\epsilon$-accuracy. Meanwhile, the space cost is $O\left(m\right)$, since only $O\left(m\right)$ variables are stored. The corresponding pseudo-code is shown in Algorithm 1:

Algorithm 1 FFT-CBA for squared-error distortion (1D)

Input:  Source distribution $p_i$, target distortion D, maximum number of iterations $max\_iter$ Output:  Minimal value ${\sum }_{i=1}^m{\sum }_{j=1}^m{p}_i{w}_{ij}\log {\displaystyle \frac{w_{ij}}{r_j}}$   1: Initialization: $\mathit{r}={\mathbf{1}}_m/m$, $\beta =0$   2: for  $t=1:max\_iter$ do   3:        while $\left|G\right(\beta \left)\right|\ge \epsilon$ do   4:             Calculate $K^{\left(0\right)},K^{\left(2\right)},K^{\left(4\right)}$ and $K_d^{\left(0\right)},K_d^{\left(2\right)},K_d^{\left(4\right)}$   5:             Calculate ${\widehat{K}}^{\left(q\right)}=\mathrm{FFT}\left(K^{\left(q\right)}\right)$, $q=0,2,4$   6:             $a=FFT\_conv(K^{\left(0\right)},r)$   7:             $u=FFT\_conv(K^{\left(2\right)},r)$   8:             $v=FFT\_conv(K^{\left(4\right)},r)$   9:             $G\leftarrow {\sum }_i{p}_i{u}_i/a_i-D$,    $G^{\prime }\leftarrow {\sum }_i{p}_i(u_i^2-a_i{v}_i)/a_i^2$ 10:             $\beta \leftarrow \beta -G/G^{\prime }$ 11:        $a=FFT\_conv(K^{\left(0\right)},r)$ 12:        $b\leftarrow p./a$ 13:        $s=FFT\_conv(K^{\left(0\right)},b)$ 14:        $r\leftarrow r\odot s$

After obtaining the optimal variables $\beta$ and $\mathit{r}$ via iterations (11), we can calculate the rate R for a given target distortion D directly:

$$R=-\beta D-\sum _{i=1}^m{p}_i\log \left(\sum _{j=1}^m{K}_{ij}{r}_j\right).$$

Here, we can apply an additional FFT to compute R, allowing us to obtain R using only $O\left(m\log m\right)$ arithmetic operations.

Remark 2.

In the implementation, the matrices $K^{\left(0\right)},K^{\left(2\right)},K^{\left(4\right)}$ do not need to be computed explicitly. Instead, we first convert the distance matrix $D_{\rho }$ into vector form and then treat this vector as a constant that is combined with the iteration-dependent β via vector-wise operations in each iteration.

Next, we consider the case of multiple dimensions. Let $\mathcal{X}=\mathcal{Y}={\mathbb{R}}^d$ with $\rho (\mathit{x},\mathit{y})={\parallel \mathit{x}-\mathit{y}\parallel }_2^2$. Under a uniform grid of side M per dimension, $m=M^d$, we have

$${\rho }_{\mathit{i}\mathit{j}}=h^2\sum _{k=1}^d{(i_k-j_k)}^2,K_{\mathit{i}\mathit{j}}=e^{-\beta h^2{\sum }_{k=1}^d{(i_k-j_k)}^2}=\prod _{k=1}^d{e}^{-\beta h^2{(i_k-j_k)}^2}.$$

Note that each convolution $K\ast v$ can be decomposed into d successive one-dimensional FFT-based convolutions:

$$K\ast v=K^{\left(1\right)}{\ast }_1\left(K^{\left(2\right)}{\ast }_2(\cdots \left(K^{\left(d\right)}{\ast }_{d}v\right)\cdots )\right),$$

where ${\ast }_k$ denotes the one-dimensional convolution along the k-th coordinate axis, and $K_{i_k{j}_k}^{\left(k\right)}=\exp \left(-\beta h^2{(i_k-j_k)}^2\right).$ Therefore, the arithmetic operations per iteration are $O\left(m\log M\right)$. Similarly to the analysis of the one-dimensional case, the complexity of the accelerated CBA algorithm under squared-error distortion with dimension d is

$$O\left({\displaystyle \frac{m\log m\log m}{\epsilon }}(1+\log |\log \epsilon \left|\right)\right)=O\left({\displaystyle \frac{{\left|\log \epsilon \right|}^2}{{\epsilon }^{d+1}}}(1+\log |\log \epsilon \left|\right)\right),$$

where $m=M^d\sim {\epsilon }^{-d}$.

Example 1.

We evaluate our accelerated algorithms on one-dimensional continuous sources and compare them with the baseline of the CBA algorithm without acceleration. We consider two classical examples—Gaussian and Laplacian sources—and evaluate each source under squared-error distortion measures. We set the accuracy to $\epsilon =0.001$, yielding $m=n=1000$ discretization nodes for X and Y, and we cap the number of iterations at $max\_iter=1000$. The discretization nodes are uniformly sampled from the interval $[-10,10]$.

Under the squared-error distortion setting, the computational time and the discrepancy in the output rate R between the two algorithms for various values of β are reported in Table 1. As shown in the table, We can see that the Fast CBA algorithm has an overwhelming advantage in computational speed. Moreover, the rates R obtained by the two algorithms are almost identical.

Table 1. Comparison between FFT-based Fast CBA and CBA algorithms under squared-error distortion.

Source	(D, R, Slope $\mathit{\beta }$)	Time (s)		Ratio	Error
		${\mathit{t}}_{\mathit{Fast}}$	${\mathit{t}}_{\mathit{CB}\mathit{A}}$	Speed-Up	$|{\mathit{R}}_{\mathit{CBA}}-{\mathit{R}}_{\mathit{Fast}}|$
Gaussian	$(0.1,1.1513,5.0000)$	0.6274	27.7583	44.2417	$9.7700\times {10}^{-15}$
	$(0.2,0.8042,2.5000)$	0.6291	30.1018	47.8485	$9.1938\times {10}^{-15}$
	$(0.3,0.6020,1.6667)$	0.6528	30.9772	47.4542	$9.4369\times {10}^{-15}$
	$(0.4,0.4581,1.2500)$	0.6600	31.1232	47.1533	$3.6082\times {10}^{-15}$
Laplacian	$(0.1,0.7343,4.9682)$	1.5500	51.9308	33.5044	$5.8176\times {10}^{-14}$
	$(0.2,0.3932,2.4218)$	1.5995	56.6702	35.4304	$4.2355\times {10}^{-14}$
	$(0.3,0.2014,1.5225)$	1.7628	57.2084	32.4523	$8.5765\times {10}^{-15}$
	$(0.4,0.0766,1.0040)$	1.5863	56.7330	35.7645	$3.1346\times {10}^{-15}$

Notes: We take $\sigma =1$ for Gaussian source $p\left(x\right)={\displaystyle \frac{1}{\sqrt{2\pi {\sigma }^2}}}{e}^{-{\displaystyle \frac{x^2}{2{\sigma }^2}}}$ and $b=0.5$ for Laplacian source $p\left(x\right)={\displaystyle \frac{1}{2b}}{e}^{-{\displaystyle \frac{\left|x\right|}{b}}}$.

Fixing $D=0.4$, by data fitting, we can see that the empirical complexity of the Fast CBA algorithm is $O\left(n^{0.86}\right)$, while that of the CBA algorithm is $O\left(n^{2.08}\right)$; see Figure 2 for illustration.

Figure 2. The comparison of computational time between the Fast CBA algorithm and the CBA algorithm with different numbers of grid points n under square-error distortion.

4.2. Absolute-Error Distortion

4.2.1. One-Dimension Case

First, we consider the $d=1$ case $\mathcal{X}=\mathcal{Y}=\mathbb{R}$ with distortion measure $\rho (x,y)=|x-y|$. Under a uniform discretization with step size h, this yields ${\rho }_{ij}=|x_i-y_j|=|i-j|h$. For convenience, we denote $\lambda =e^{-\beta h}$.

Substituting (10b) into (10c), the iterations of the CBA algorithm become:

$$\begin{array}{cc}\hfill \mathrm{Solve}G\left(\beta \right)& \triangleq \sum _{i=1}^m{\displaystyle \frac{\left({\sum }_{j=1}^m{\lambda }^{|i-j|}|i-j|h{r}_j\right)}{\left({\sum }_{j=1}^m{\lambda }^{|i-j|}{r}_j\right)}}{p}_i-D=0,\hfill \end{array}$$

(14a)

$$\begin{array}{cc}\hfill r_j& =r_j\sum _{i=1}^m{\displaystyle \frac{p_i{\lambda }^{|i-j|}}{{\sum }_{j=1}^m{\lambda }^{|i-j|}{r}_j}},j=1,2\cdots m.\hfill \end{array}$$

(14b)

Although the equation $G\left(\beta \right)=0$ cannot be solved exactly, the prescribed $\epsilon$-accuracy of the final result is guaranteed, as established in [12] (Theorem 3.2). In practice, Newton’s method for solving (14a) is implemented via the following iterative scheme until the stopping criterion $\left|G\right(\beta \left)\right|\le \epsilon$ is satisfied.

$$\beta \leftarrow \beta -G\left(\beta \right)/G^{\prime }\left(\beta \right).$$

(15)

By calculation, $G^{\prime }\left(\beta \right)$ can be written as the following form:

$$G^{\prime }\left(\beta \right)=\sum _{i=1}^m{p}_i{\displaystyle \frac{-\left({\sum }_{j=1}^m{\lambda }^{|i-j|}{|i-j|}^2{h}^2{r}_j\right)\left({\sum }_{j=1}^m{\lambda }^{|i-j|}{r}_j\right)+{\left({\sum }_{j=1}^m{\lambda }^{|i-j|}|i-j|h{r}_j\right)}^2}{{\left({\sum }_{j=1}^m{\lambda }^{|i-j|}{r}_j\right)}^2}}.$$

Therefore, the main arithmetic complexity in the update of $\beta$ lies in the computation of ${\sum }_{j=1}^m{\lambda }^{|i-j|}|i-j|r_j$ and ${\sum }_{j=1}^m{\lambda }^{|i-j|}{|i-j|}^2{r}_j$.

Since $\lambda$ is related to $\beta$, we denote $a_i={\sum }_{j=1}^m{\lambda }^{|i-j|}{r}_j$. Let

$$b_i=\sum _{j=1}^i{\lambda }^{|i-j|}{r}_j,c_i=\sum _{j=i+1}^m{\lambda }^{|i-j|}{r}_j,a_i=b_i+c_i.$$

Similarly to the treatment in [20], we have the following recursive formulae:

$$\begin{array}{cc}\hfill & b_{i+1}=\lambda b_i+r_{i+1},i=1,2\cdots m-1,\hfill \\ \hfill & c_{i-1}=\lambda (c_i+r_i),i=m,m-1\cdots 2,\hfill \end{array}$$

with initial values $b_1=r_1,c_m=0$.

By exploiting the above recursion, the sequences ${\left\{b_i\right\}}_{i=1}^m$ and ${\left\{c_i\right\}}_{i=1}^m$ can be computed sequentially. Each step requires only one multiplication and one addition. Consequently, the total number of arithmetic operations needed to compute $b_i$ and $c_i$, for $i=1,2,\cdots ,m$, is $O\left(m\right)$. Therefore, the evaluation of $a_i$ can be completed in $O\left(m\right)$ arithmetic operations.

Next, we leverage dynamic programming [32] to compute the sums ${\sum }_{j=1}^m{\lambda }^{|i-j|}|i-j|r_j$ and ${\sum }_{j=1}^m{\lambda }^{|i-j|}{|i-j|}^2{r}_j$ efficiently. Note that these sums differ from ${\sum }_{j=1}^m{\lambda }^{|i-j|}{r}_j$, so we cannot directly adopt the approach in [20]. However, with appropriate adjustments, we can also compute them in linear complexity. We denote

$$u_i=\sum _{j=1}^i{\lambda }^{|i-j|}|i-j|r_j,v_i=\sum _{j=i+1}^m{\lambda }^{|i-j|}|i-j|r_j.$$

Then,

$$\begin{array}{c}\hfill \lambda u_i=u_{i+1}-\lambda b_i,\lambda v_i=v_{i-1}-\lambda r_i-\lambda c_i.\end{array}$$

Thus, we obtain the following recursive formulae:

$$\begin{array}{cc}\hfill & u_{i+1}=\lambda (u_i+b_i),i=1,2\cdots m-1,\hfill \\ \hfill & v_{i-1}=\lambda (v_i+c_i+r_i),i=m,m-1\cdots 2,\hfill \end{array}$$

with initial values $u_1=0,v_m=0$. In each recursion, we perform three additions and two multiplications. Since ${\left\{b_i\right\}}_{i=1}^m$ and ${\left\{c_i\right\}}_{i=1}^m$ can be calculated within $O\left(m\right)$ arithmetic operations, to calculate $u_i,v_{i}i=1,2,\cdots m$, $O\left(m\right)$ arithmetic operations are needed. Then

$$\sum _{j=1}^m{\lambda }^{|i-j|}|i-j|r_j=u_i+v_i,i=1,2,\cdots m,$$

(16)

can also be calculated within $O\left(m\right)$ arithmetic operations.

Meanwhile, let

$${\zeta }_i=\sum _{j=1}^i{\lambda }^{|i-j|}{|i-j|}^2{r}_j,{\eta }_i=\sum _{j=i+1}^m{\lambda }^{|i-j|}{|i-j|}^2{r}_j.$$

Then, it holds that

$$\begin{array}{cc}\hfill \lambda {\zeta }_i& =\sum _{j=1}^i{\lambda }^{i-j+1}{(i-j)}^2{r}_j\hfill \\ \hfill & =\sum _{j=1}^i{\lambda }^{i-j+1}{(i-j+1)}^2{r}_j-2\sum _{j=1}^i{\lambda }^{i-j+1}(i-j)r_j-\sum _{j=1}^i{\lambda }^{i-j+1}{r}_j\hfill \\ \hfill & ={\zeta }_{i+1}-2\lambda u_i-\lambda b_i.\hfill \end{array}$$

Similarly,

$$\lambda {\eta }_i={\eta }_{i-1}-\lambda r_i-2\lambda v_i-\lambda c_i.$$

Thus, we obtain the following recursive formulae:

$$\begin{array}{cc}\hfill & {\zeta }_{i+1}=\lambda ({\zeta }_i+2u_i+b_i),i=1,2\cdots m-1,\hfill \\ \hfill & {\eta }_{i-1}=\lambda ({\eta }_i+2v_i+c_i+r_i),i=m,m-1\cdots 2,\hfill \end{array}$$

with initial values ${\zeta }_1=0,{\eta }_m=0$. In each recursion, we perform five additions and four multiplications. Hence, to calculate ${\zeta }_i,{\eta }_{i}i=1,2,\cdots m$, $O\left(m\right)$ arithmetic operations are needed. Then

$$\sum _{j=1}^m{\lambda }^{|i-j|}{|i-j|}^2{r}_j={\zeta }_i+{\eta }_i,i=1,2,\cdots m,$$

can also be calculated within $O\left(m\right)$ arithmetic operations.

Since ${\sum }_{j=1}^m{\lambda }^{|i-j|}{r}_j,{\sum }_{j=1}^m{\lambda }^{|i-j|}|i-j|r_j,{\sum }_{j=1}^m{\lambda }^{|i-j|}{|i-j|}^2{r}_j,i=1,2,\cdots m$ can be calculated within $O\left(m\right)$ arithmetic operations, we can use only $O\left(m\right)$ arithmetic operations to calculate $G^{\prime }\left(\beta \right)$. Similarly, $G\left(\beta \right)$ can also be calculated within $O\left(m\right)$ arithmetic operations. Thus, each Newton’s iteration involves $O\left(m\right)$ arithmetic operations. Due to the locally quadratic convergence of the Newton’s method, we need $\log \left|\log \epsilon \right|$ iterations to achieve $\epsilon$-accuracy. Hence, we only need $O\left(m\log \right|\log \epsilon \left|\right)$ arithmetic operations to implement (14a). Notably, the iteration (14b) can be divided into two steps:

$$\begin{array}{cc}\hfill & a_i\leftarrow \sum _{j=1}^m{\lambda }^{|i-j|}{r}_j,i=1,2\cdots m,r_j\leftarrow r_j\sum _{i=1}^m{\displaystyle \frac{p_i{\lambda }^{|i-j|}}{a_i}},j=1,2\cdots m.\hfill \end{array}$$

(17)

Based on the above analysis, each iteration of (17) can be carried out with linear computational complexity $O\left(m\right)$.

Therefore, in each iteration of (14), it involves $O\left(m(1+\log |\log \epsilon \left|\right)\right)$ arithmetic operations. Due to [12] (Theorem 3.2), we need $O(\log m/\epsilon )$ iterations. Thus, the total computational cost of our method is $O\left({\displaystyle \frac{m\log m}{\epsilon }}(1+\log |\log \epsilon \left|\right)\right)$. Since $m\sim 1/\epsilon$ to ensure the accuracy of numerical integration, the computational cost turns out to be $O\left({\displaystyle \frac{\left|\log \epsilon \right|}{{\epsilon }^2}}(1+\log |\log \epsilon \left|\right)\right)$ to achieve $\epsilon$-accuracy. Meanwhile, the space cost is $O\left(m\right)$, since only $O\left(m\right)$ variables are stored. The corresponding pseudo-code is shown in Algorithm 2.

Algorithm 2 Accelerated CBA algorithm for absolute-error distortion (1D)

Input:  Source distribution $p_i$, target distortion D, maximum number of iterations $max\_iter$ Output:  Minimal value ${\sum }_{i=1}^m{\sum }_{j=1}^m{p}_i{w}_{ij}\log {\displaystyle \frac{w_{ij}}{r_j}}$   1: Initialization:  $\mathit{r}={\mathbf{1}}_m/m,\beta =0$   2: for$\ell =1:max\_iter$ do   3:       while $\left|G\right(\beta \left)\right|\ge \epsilon$ do   4:             $\lambda \leftarrow e^{-\beta h}$   5:             $b_1\leftarrow r_1,c_m\leftarrow 0,u_1\leftarrow 0,v_m\leftarrow 0,{\zeta }_1\leftarrow 0,{\eta }_m\leftarrow 0$   6:             for $i=1:m-1$ do   7:                   $b_{i+1}\leftarrow \lambda b_i+r_{i+1}$   8:                   $c_{m-i}\leftarrow \lambda (c_{m-i+1}+r_{m-i+1})$   9:                   $u_{i+1}\leftarrow \lambda (u_i+b_i)$ 10:                   $v_{m-i}\leftarrow \lambda (v_{m-i+1}+c_{m-i+1}+r_{m-i+1})$ 11:                   ${\zeta }_{i+1}\leftarrow \lambda ({\zeta }_i+2u_i+b_i)$ 12:                   ${\eta }_{m-i}\leftarrow \lambda ({\eta }_{m-i+1}+2v_{m-i+1}+c_{m-i+1}+r_{m-i+1})$ 13:             $\beta \leftarrow \beta -\left({\sum }_{i=1}^m{p}_{i}h{\displaystyle \frac{(u_i+v_i)}{(b_i+c_i)}}-D\right)/\left({\sum }_{i=1}^m{p}_i{h}^2{\displaystyle \frac{-({\zeta }_i+{\eta }_i)(b_i+c_i)+{(u_i+v_i)}^2}{{(b_i+c_i)}^2}}\right)$ 14:       $a_i\leftarrow b_i+c_i$ 15:       $b_1^{\prime }\leftarrow p_1/a_1,c_m^{\prime }\leftarrow 0$ 16:       for $j=1:m-1$ do 17:             $b_{j+1}^{\prime }\leftarrow \lambda b_j^{\prime }+p_{j+1}/a_{j+1}$ 18:             $c_{m-j}^{\prime }\leftarrow \lambda (c_{m-j+1}^{\prime }+p_{m-j+1}/a_{m-j+1})$ 19:       $r_j\leftarrow r_j(b_j^{\prime }+c_j^{\prime })$

After obtaining the optimal variables $\beta$ and $\mathit{r}$ via iterations (14), we can calculate the rate R for given target distortion D directly:

$$\begin{array}{cc}\hfill R=I(X;Y)& =\sum _{i=1}^m\sum _{j=1}^m{p}_i{w}_{ij}\log {\displaystyle \frac{w_{ij}}{r_j}}=\sum _{i=1}^m\sum _{j=1}^m{p}_i{w}_{ij}\log {\displaystyle \frac{e^{-\beta {\rho }_{ij}}}{{\sum }_{j=1}^m{e}^{-\beta {\rho }_{ij}}{r}_j}}\hfill \\ \hfill & =-\beta \sum _{i=1}^m\sum _{j=1}^m{p}_i{w}_{ij}{\rho }_{ij}-\sum _{i=1}^m{p}_i\log \sum _{j=1}^m{e}^{-\beta {\rho }_{ij}}{r}_j=-\beta D-\sum _{i=1}^m{p}_i\log \sum _{j=1}^m{\lambda }^{|i-j|}{r}_j.\hfill \end{array}$$

(18)

Here, we have used the condition ${\sum }_j{w}_{ij}=1$. Since ${\sum }_{j=1}^m{\lambda }^{|i-j|}{r}_j,i=1,2,\cdots m$ can be calculated within $O\left(m\right)$ arithmetic operations, we only need $O\left(m\right)$ arithmetic operations to obtain R.

Example 2.

We evaluate our accelerated algorithms on one-dimensional continuous sources and compare them with the baseline of the CBA algorithm without acceleration. We consider two classical examples—Gaussian and Laplacian sources—and evaluate each source under absolute-error distortion measures. We set the accuracy to $\epsilon =0.001$, yielding $m=n=1000$ discretization nodes for X and Y, and we cap the number of iterations at $max\_iter=1000$. The discretization nodes are uniformly sampled from the interval $[-10,10]$.

Under the absolute-error distortion setting, the computational time and the discrepancy in the output rate R between the two algorithms for different values of β are reported in Table 2. As shown in the table, We can see that the Fast CBA algorithm has an overwhelming advantage in computational speed. Moreover, the rate R obtained by the two algorithms are almost identical.

Table 2. Comparison between Fast CBA and CBA algorithms under absolute-error distortion.

Source	(D, R, Slope $\mathit{\beta }$)	Time (s)		Ratio	Error
		${\mathit{t}}_{\mathit{Fast}}$	${\mathit{t}}_{\mathit{CB}\mathit{A}}$	Speed-Up	$|{\mathit{R}}_{\mathit{CBA}}-{\mathit{R}}_{\mathit{Fast}}|$
Gaussian	$(0.1,2.0251,9.9345)$	0.0394	74.7331	$1.8945\times {10}^3$	$8.7953\times {10}^{-11}$
	$(0.2,1.3344,4.9917)$	0.0425	80.5380	$1.8961\times {10}^3$	$1.2735\times {10}^{-10}$
	$(0.3,0.9294,3.3303)$	0.0771	138.1042	$1.7916\times {10}^3$	$1.3441\times {10}^{-9}$
	$(0.4,0.6422,2.4901)$	0.0752	140.3673	$1.8667\times {10}^3$	$1.9662\times {10}^{-8}$
Laplacian	$(0.1,1.6062,9.9345)$	0.0761	156.8812	$2.0616\times {10}^3$	$7.6961\times {10}^{-9}$
	$(0.2,0.9155,4.9916)$	0.0748	151.2206	$2.0210\times {10}^3$	$4.9796\times {10}^{-8}$
	$(0.3,0.5106,3.3307)$	0.0762	148.8361	$1.9539\times {10}^3$	$1.2569\times {10}^{-7}$
	$(0.4,0.2231,2.4986)$	0.0767	150.7649	$1.9655\times {10}^3$	$2.2397\times {10}^{-7}$

Notes: We take $\sigma =1$ for Gaussian source $p\left(x\right)={\displaystyle \frac{1}{\sqrt{2\pi {\sigma }^2}}}{e}^{-{\displaystyle \frac{x^2}{2{\sigma }^2}}}$ and $b=0.5$ for Laplacian source $p\left(x\right)={\displaystyle \frac{1}{2b}}{e}^{-{\displaystyle \frac{\left|x\right|}{b}}}$.

Fixing $D=0.4$, by data fitting, we can see that the empirical complexity of the Fast CBA algorithm is $O\left(n^{1.07}\right)$, while that of the CBA algorithm is $O\left(n^{2.13}\right)$; see Figure 3 for illustration.

Figure 3. The comparison of computational time between the Fast CBA algorithm and the CBA algorithm with different numbers of grid points n under absolute-error distortion.

4.2.2. Extension to Higher Dimensions

Next, we consider the case $\mathcal{X}=\mathcal{Y}={\mathbb{R}}^d$ with the distortion measure $\rho (x,y)={\parallel \mathit{x}-\mathit{y}\parallel }_1={\sum }_{i=1}^d|x^i-y^i|$, where $\mathit{x}=(x^1,x^2,\cdots ,x^d),\mathit{y}=(y^1,y^2,\cdots ,y^d)\in {\mathbb{R}}^d$. Thus, after discretization, ${\rho }_{\mathit{i}\mathit{j}}={\sum }_{k=1}^d|x_{i_k}^k-y_{j_k}^k|={\sum }_{k=1}^d|i_k-j_k|h,i_1,i_2,\cdots ,i_d,j_1,j_2,\cdots j_d=1,2,\cdots M$ and $\mathit{i}=(i_1,i_2,\cdots ,i_d),\mathit{j}=(j_1,j_2,\cdots ,j_d)$. The total number of discretization nodes is $m=M^d$. Here, we adopt the uniform discretization with step size h. For simplicity, let $\lambda =e^{-\beta h}$, $|\mathit{i}-\mathit{j}|={\sum }_{k=1}^d|i_k-j_k|$, and ${\sum }_{\mathit{i}}$ be ${\sum }_{i_1,i_2\cdots ,i_d=1}^M$, i.e., taking the summation over all indices $i_1,i_2,\cdots ,i_d$. Then, $e^{-\beta {\rho }_{\mathit{i}\mathit{j}}}={\prod }_{k=1}^d{\lambda }^{|i_k-j_k|}={\lambda }^{|\mathit{i}-\mathit{j}|}$.

Similarly to the one-dimensional case, we only need to accelerate the calculation of (14a) and (14b) with dimension d. For (14b),

$$\begin{array}{cc}\hfill \sum _{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}{r}_{\mathit{j}}& =\sum _{j_1,\cdots j_d=1}^M{\lambda }^{|i_1-j_1|}\cdots {\lambda }^{|i_d-j_d|}{r}_{j_1\cdots j_d}\hfill \\ \hfill & =\sum _{j_1,\cdots j_{d-1}=1}^M{\lambda }^{|i_1-j_1|}\cdots {\lambda }^{|i_{d-1}-j_{d-1}|}\sum _{j_d=1}^M{\lambda }^{|i_d-j_d|}{r}_{j_1\cdots j_d}.\hfill \end{array}$$

By fixing $j_1,\cdots ,j_{d-1}$, we need $O\left(M\right)$ arithmetic operations to calculate ${\tilde{r}}_{i_d,j_1\cdots j_{d-1}}={\sum }_{j_d=1}^M{\lambda }^{|i_d-j_d|}{r}_{j_1\cdots j_d},i_d=1,2\cdots M$. Thus, ${\tilde{r}}_{i_d,j_1\cdots j_{d-1}},i_d,j_1,\cdots j_{d-1}=1,2\cdots M$ can be calculated within $O\left(M^d\right)=O\left(m\right)$ arithmetic operations. Then,

$$\begin{array}{c}\hfill \sum _{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}{r}_{\mathit{j}}=\sum _{j_1,\cdots j_{d-1}=1}^M{\lambda }^{|i_1-j_1|}\cdots {\lambda }^{|i_{d-1}-j_{d-1}|}{\tilde{r}}_{i_d,j_1\cdots j_{d-1}},i_1,\cdots i_{d-1}=1,2\cdots M,\end{array}$$

which is exactly the case for dimension $d-1$ when fixing $i_d$. Set $T\left(k\right)$ as the complexity to calculate ${\sum }_{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}{r}_{\mathit{j}}$ for dimension k. Then, the recursive formula for $T\left(k\right)$ becomes:

$$T\left(k\right)=O\left(M^k\right)+M·T(k-1).$$

Since $T\left(1\right)$ is the complexity of calculating ${\sum }_{j=1}^M{\lambda }^{|i-j|}{r}_j$, we have $T\left(1\right)=O\left(M\right)$. Therefore, $T\left(d\right)=O\left(Md\right)=O\left(m\right)$.

For (14a), the main arithmetic complexity lies in the calculation of ${\sum }_{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}|\mathit{i}-\mathit{j}|r_{\mathit{j}}$ and ${\sum }_{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}{|\mathit{i}-\mathit{j}|}^2{r}_{\mathit{j}}$.

Note that

$$\begin{array}{cc}\hfill \sum _{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}|\mathit{i}-\mathit{j}|r_{\mathit{j}}& =\sum _{k=1}^d\sum _{j_1,\cdots j_d=1}^M{\lambda }^{|i_1-j_1|}\cdots {\lambda }^{|i_d-j_d|}|i_k-j_k|r_{j_1\cdots j_d}\hfill \\ \hfill & =\sum _{k=1}^d\sum _{j_1\cdots j_{k-1},j_{k+1}\cdots j_d=1}^M\prod _{s\ne k}{\lambda }^{|i_s-j_s|}\sum _{j_k=1}^M{\lambda }^{|i_k-j_k|}|i_k-j_k|r_{j_1\cdots j_d}.\hfill \end{array}$$

Fixing $j_1\cdots j_{k-1},j_{k+1}\cdots j_d$, we can then calculate ${\tilde{r}}_{i_k,j_1\cdots j_{k-1},j_{k+1}\cdots j_d}={\sum }_{j_k=1}^M{\lambda }^{|i_k-j_k|}|i_k-j_k|r_{j_1\cdots j_d},i_k=1,2\cdots M$ within $O\left(M\right)$ arithmetic operations. Furthermore, ${\tilde{r}}_{i_k,j_1\cdots j_{k-1},j_{k+1}\cdots j_d}$, $i_k,j_1\cdots j_{k-1},j_{k+1}\cdots j_d=1,2\cdots M$ can be calculated within $O\left(M^d\right)=O\left(m\right)$ arithmetic operations. Meanwhile, ${\sum }_{j_1\cdots j_{k-1},j_{k+1}\cdots j_d=1}^M{\prod }_{s\ne k}{\lambda }^{|i_s-j_s|}{\tilde{r}}_{i_k,j_1\cdots j_{k-1},j_{k+1}\cdots j_d}$ can be calculated within $O\left(m\right)$ arithmetic operations due to the aforementioned analysis. Thus, $O\left(dm\right)=O\left(m\right)$ arithmetic operations are needed to calculate ${\sum }_{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}|\mathit{i}-\mathit{j}|r_{\mathit{j}}$.

In the meantime,

$$\begin{array}{cc}\hfill & \sum _{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}{|\mathit{i}-\mathit{j}|}^2{r}_{\mathit{j}}=\sum _{k,\ell =1}^d\sum _{j_1,\cdots j_d=1}^M{\lambda }^{|i_1-j_1|}\cdots {\lambda }^{|i_d-j_d|}|i_k-j_k||i_{\ell }-j_{\ell }|r_{j_1\cdots j_d}\hfill \\ \hfill & =\sum _{k\ne \ell }\sum _{j_1\cdots j_{k-1},j_{k+1}\cdots j_{\ell -1},j_{\ell +1}\cdots j_d=1}^M\prod _{s\ne k,\ell }{\lambda }^{|i_s-j_s|}\sum _{j_{\ell }=1}^M{\lambda }^{|i_{\ell }-j_{\ell }|}|i_{\ell }-j_{\ell }|\sum _{j_k=1}^M{\lambda }^{|i_k-j_k|}|i_k-j_k|r_{j_1\cdots j_d}\hfill \\ \hfill & +\sum _{k=1}^d\sum _{j_1\cdots j_{k-1},j_{k+1}\cdots j_d=1}^M\prod _{s\ne k}{\lambda }^{|i_s-j_s|}\sum _{j_k=1}^M{\lambda }^{|i_k-j_k|}{|i_k-j_k|}^2{r}_{j_1\cdots j_d}.\hfill \end{array}$$

By similarly fixing $j_1,\cdots ,j_{k-1},j_{k+1},\cdots ,j_d$, each component of ${\sum }_{\mathit{j}}{\lambda }^{|\mathit{i}-\mathit{j}|}{|\mathit{i}-\mathit{j}|}^2{r}_{\mathit{j}}$ can be computed with $O\left(m\right)$ arithmetic operations. Therefore, we obtain the estimate of computational complexity $O\left({\displaystyle \frac{\left|\log \epsilon \right|}{{\epsilon }^{d+1}}}(1+\log |\log \epsilon \left|\right)\right)$ to achieve $\epsilon$-accuracy of our method. Moreover, the rate R can also be calculated using $O\left(m\right)=O(1/{\epsilon }^d)$ arithmetic operations.

5. Conclusions

In this article, we conducted an integrated approach for computing rate–distortion functions, connecting the RD problem formulation of continuous sources and discretization schemes along with their corresponding efficient discrete algorithms. We first showed that discretized reproduction distributions provably converge to the solution of the underlying continuous RD problem, and derived quantitative error bounds that clarify the impact of discretization resolution and problem dimension. In the discrete setting, we revisited the Blahut–Arimoto (BA) and constrained BA (CBA) algorithms, establishing their arithmetic complexity properties for achieving $ϵ$-accuracy guarantees when used, together with discretization, as solvers for the underlying continuous RD problem. For certain distortion measures, we further designed acceleration techniques that exploit the structure in discretized problems, including squared-error and absolute-error distortion measures.

Appendix A.1. Proof of Lemma 1

Proof. 

We denote the optimal solution of (5) as ${\mathit{r}}^{*}$ and $W_n=\left\{{\sum }_{j=1}^n{r}_j{\delta }_{y_j^n}\right|r_j\ge 0,{\sum }_{j=1}^n{r}_j=1\}$. As shown in Section 2.1, the continuous RD problem has the discrete form (6), which is in fact minimizing $f\left(\mathit{r}\right)$ as defined in (5), but over $W_n$ instead of $\mathcal{W}$.

Denote the optimal solution of the discrete problem as ${\mathit{r}}^n$. Then $f\left({\mathit{r}}^n\right)\le f\left(\mathit{r}\right),\forall \mathit{r}\in W_n$. Notice that if there exists ${\tilde{\mathit{q}}}^n\in W_n$ satisfying $f\left({\tilde{\mathit{q}}}^n\right)\to f\left({\mathit{r}}^{*}\right)$, then

$$f\left({\mathit{r}}^{*}\right)\le f\left({\mathit{r}}^n\right)\le f\left({\tilde{\mathit{q}}}^n\right)\to f\left({\mathit{r}}^{*}\right).$$

(A1)

Here, the first inequality is due to the fact $W_n\subseteq \mathcal{W}$. The relationship (A1) would then lead to the convergence $f\left({\mathit{r}}^n\right)\to f\left({\mathit{r}}^{*}\right)$. Now, let ${\mathit{q}}^n={\sum }_{j=1}^n{A}_j^n{\delta }_{y_j^n}$, where $A_j^n={\int }_{I_j}\mathrm{d}{\mathit{r}}^{*}$ and $I_j$ is defined in Section 2.1. Let ${\tilde{\mathit{q}}}^n={\mathit{q}}^n/\int \mathrm{d}{\mathit{q}}^n$. Since $\int \mathrm{d}{\mathit{q}}^n={\sum }_j{\int }_{I_j}\mathrm{d}{\mathit{r}}^{*}\le 1$, one has

$$f\left({\tilde{\mathit{q}}}^n\right)=f\left({\mathit{q}}^n\right)+\log (\int \mathrm{d}{\mathit{q}}^n)\le f\left({\mathit{q}}^n\right).$$

Since ${\tilde{\mathit{q}}}^n\in W_n$ and $f\left({\tilde{\mathit{q}}}^n\right)\le f\left({\mathit{q}}^n\right)$, it suffices to show that $f\left({\mathit{q}}^n\right)\to f\left({\mathit{r}}^{*}\right)$. Before doing so, convergence under a truncation of $\mathcal{X}$ needs to be established. Choose $A>1$ such that ${\int }_{\parallel y\parallel \le A-1}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\ge \frac{1}{2}$, and fix any $M\ge A$. First, for $x\in {[-M,M]}^d$ and sufficiently large n, the following estimate holds:

$$\begin{array}{cc}\hfill & |\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)-\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)|\hfill \\ \hfill & \le \sum _i{\int }_{I_i}|\exp (-\beta \rho (x,y))-\exp (-\beta \rho (x,y_i^n))|\mathrm{d}{\mathit{r}}^{*}\left(y\right)+{\int }_{\left|y\right|>n^{{\displaystyle \frac{1}{2d}}}}\exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)\hfill \\ \hfill & \le \sum _i{\int }_{I_i}\epsilon \mathrm{d}{\mathit{r}}^{*}\left(y\right)+{\int }_{\left|y\right|>n^{{\displaystyle \frac{1}{2d}}}}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\le 2\epsilon ,\hfill \end{array}$$

where the second inequality invokes the assumption together with the decomposition ${\mathit{q}}^n={\sum }_j{\int }_{I_j}\mathrm{d}{\mathit{r}}^{*}{\delta }_{y_j^n}$. Next, we establish the uniform convergence. For $x\in {[-M,M]}^d$,

$$\begin{array}{cc}\hfill & \int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\ge {\int }_{\parallel y\parallel \le M}\exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\hfill \\ \hfill & \ge {\int }_{\parallel y\parallel \le M}\exp (-\beta {\rho }^{*})\mathrm{d}{\mathit{q}}^n\left(y\right)\ge {\displaystyle \frac{1}{2}}\exp (-\beta {\rho }^{*})\triangleq \delta .\hfill \end{array}$$

(A2)

Here ${\rho }^{*}={\max }_{x,y\in {[-M,M]}^d}\rho (x,y)$ and ${\int }_{\parallel y\parallel \le M}\mathrm{d}{q}^n\ge {\displaystyle \frac{1}{2}}$ is due to

$$\begin{array}{cc}\hfill {\int }_{\parallel y\parallel \le A}\mathrm{d}{\mathit{q}}^n\left(y\right)& =\sum _{i:y_i^n\in {[-A,A]}^d}{\int }_{I_i}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\ge {\int }_{\parallel y\parallel \le A-1}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\ge {\displaystyle \frac{1}{2}}.\hfill \end{array}$$

(A3)

Since $\log x$ is uniformly continuous over $[\delta ,+\infty )$,

$$\begin{array}{c}\hfill \log \left(\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\right)\rightrightarrows \log \left(\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right),x\in {[-M,M]}^d.\end{array}$$

Then, for sufficiently large n,

$$\begin{array}{cc}\hfill & |{\int }_{\parallel x\parallel \le M}\log \left[\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\right]\mathrm{d}p\left(x\right)\hfill \\ \hfill & -{\int }_{\parallel x\parallel \le M}\log \left[\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right]\mathrm{d}p\left(x\right)|\le {\int }_{\parallel x\parallel \le M}\epsilon \mathrm{d}p\left(x\right)\le \epsilon .\hfill \end{array}$$

As $n\to +\infty$,

$$\begin{array}{c}{\int }_{\parallel x\parallel \le M}\log \left[\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\right]\mathrm{d}p\left(x\right)\to \hfill \\ {\int }_{\parallel x\parallel \le M}\log \left[\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right]\mathrm{d}p\left(x\right).\hfill \end{array}$$

Hence, convergence after truncation holds. It remains to prove that $f\left({\mathit{q}}^n\right)\to f\left({\mathit{r}}^{*}\right)$. For $M\ge A$,

$$\begin{array}{cc}\hfill & |f\left({\mathit{q}}^n\right)-f\left({\mathit{r}}^{*}\right)|=|\int {\mathcal{H}}_{{\mathit{q}}^n\left(y\right)}\left(x\right)\mathrm{d}p\left(x\right)-\int {\mathcal{H}}_{{\mathit{r}}^{*}\left(y\right)}\left(x\right)\mathrm{d}p\left(x\right)|\hfill \\ \hfill & \le \left|{\int }_{\parallel x\parallel \le M}[{\mathcal{H}}_{{\mathit{q}}^n\left(y\right)}\left(x\right)-{\mathcal{H}}_{{\mathit{r}}^{*}\left(y\right)}\left(x\right)]\mathrm{d}p\left(x\right)\right|+{\int }_{\parallel x\parallel >M}\left(-{\mathcal{H}}_{{\mathit{q}}^n\left(y\right)}\left(x\right)-{\mathcal{H}}_{{\mathit{r}}^{*}\left(y\right)}\left(x\right)\right)\mathrm{d}p\left(x\right),\hfill \end{array}$$

(A4)

where ${\mathcal{H}}_{\mathit{r}\left(y\right)}\left(x\right)=\log \int \exp (-\beta \rho (x,y))\mathrm{d}\mathit{r}\left(y\right)$. Let $g\left(x\right)={\max }_{\parallel y\parallel \le A}\rho (x,y)$. Note that

$$\begin{array}{cc}\hfill \int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)& \ge \exp (-\beta g\left(x\right)){\int }_{\parallel y\parallel \le A}\mathrm{d}{\mathit{q}}^n\left(y\right)\ge {\displaystyle \frac{1}{2}}\exp (-\beta g\left(x\right)),\hfill \end{array}$$

where the second inequality is similar to (A3). Similarly, we have

$$\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)\ge {\displaystyle \frac{1}{2}}\exp (-\beta g\left(x\right)).$$

Thus, combined with the assumption, we get

$$\begin{array}{cc}\hfill & {\int }_{\parallel x\parallel >M}\left(-{\mathcal{H}}_{{\mathit{q}}^n}\left(x\right)-{\mathcal{H}}_{{\mathit{r}}^{*}}\left(x\right)\right)\mathrm{d}p\left(x\right)\le {\int }_{\parallel x\parallel >M}(2\log 2+2\beta g\left(x\right))\mathrm{d}p\left(x\right)\to 0,M\to +\infty .\hfill \end{array}$$

Therefore, by (A4) in the proof of Lemma 2,

$$\begin{array}{cc}\hfill |f\left({\mathit{q}}^n\right)-f\left({\mathit{r}}^{*}\right)|& \le \left|{\int }_{\parallel x\parallel \le M}[{\mathcal{H}}_{{\mathit{q}}^n\left(y\right)}\left(x\right)-{\mathcal{H}}_{{\mathit{r}}^{*}\left(y\right)}\left(x\right)]\mathrm{d}p\left(x\right)\right|+{\int }_{\parallel x\parallel >M}(2\log 2+2\beta g\left(x\right))\mathrm{d}p\left(x\right).\hfill \end{array}$$

By taking the upper limit of n for this inequality, we obtain

$$\begin{array}{cc}\hfill & \underset{n}{\lim \sup }|f\left({\mathit{q}}^n\right)-f\left({\mathit{r}}^{*}\right)|\le {\int }_{\parallel x\parallel >M}(2\log 2+2\beta g\left(x\right))\mathrm{d}p\left(x\right).\hfill \end{array}$$

Taking the limit $M\to +\infty$, it follows that $f\left({\mathit{q}}^n\right)\to f\left({\mathit{r}}^{*}\right)$.

To prove the convergence of the solutions, let $\tilde{\mathit{r}}$ be a limit point of the solution sequence ${\left\{{\mathit{r}}^n\right\}}_{n=1}^{\infty }$. Then there exists a subsequence ${\left\{{\mathit{r}}^{n_k}\right\}}_{k=1}^{\infty }$ satisfying ${\mathit{r}}^{n_k}\stackrel{w}{\to }\tilde{\mathit{r}}$. Next,

$$f\left(\tilde{\mathit{r}}\right)=\underset{k}{\lim }f\left({\mathit{r}}^{n_k}\right)=f\left({\mathit{r}}^{*}\right),$$

since ${\lim }_{n}f\left({\mathit{r}}^n\right)=f\left({\mathit{r}}^{*}\right)$. Therefore, we have shown that $f\left(\tilde{\mathit{r}}\right)$ is equal to the optimal value, and consequently $\tilde{\mathit{r}}$ is an optimal solution. □

Appendix A.2. Proof of Proposition 1

Proof. 

Denote $W_n=\left\{{\sum }_{j=1}^n{r}_j{\delta }_{y_j^n}\right|r_j\ge 0,{\sum }_{j=1}^n{r}_j=1\}$. Similarly to Lemma 1, the discrete problem (7) is equivalent to optimizing $F(\mathit{r},\beta )$ over $W_n$.

By the property of max–min problems (7) and (3), $\forall \mathit{r}\in W_n,\forall \beta \ge 0$,

$$F({\mathit{r}}^n,{\beta }^n)\le F(\mathit{r},{\beta }^n),F({\mathit{r}}^n,{\beta }^n)\ge F({\mathit{r}}^n,\beta ).$$

Furthermore, $\forall \mathit{r}\in W,\forall \beta \ge 0$,

$$F({\mathit{r}}^{*},{\beta }^{*})\le F(\mathit{r},{\beta }^{*}),F({\mathit{r}}^{*},{\beta }^{*})\ge F({\mathit{r}}^{*},\beta ).$$

Next, let ${\tilde{\mathit{q}}}^n={\mathit{q}}^n/\int \mathrm{d}{\mathit{q}}^n\in W_n$. Since $\int \mathrm{d}{\mathit{q}}^n\le 1$,

$$F({\tilde{\mathit{q}}}^n,\beta )=F({\mathit{q}}^n,\beta )+\log (\int \mathrm{d}{\mathit{q}}^n)\le F({\mathit{q}}^n,\beta ),\forall \beta \ge 0.$$

Consequently, the following chain of inequalities holds:

$$F({\mathit{r}}^{*},{\beta }^{*})\le F({\mathit{r}}^n,{\beta }^{*})\le F({\mathit{r}}^n,{\beta }^n)\le F({\tilde{\mathit{q}}}^n,{\beta }^n)\le F({\mathit{q}}^n,{\beta }^n)\le F({\mathit{q}}^n,{\tilde{\beta }}^n),$$

where ${\tilde{\beta }}^n={\mathrm{argmax}}_{\beta }F({\mathit{q}}^n,\beta )$ and we have used the max–min optimality property of $({\mathit{r}}^{*},{\beta }^{*})$. Note that if $F({\mathit{q}}^n,{\tilde{\beta }}^n)\to F({\mathit{r}}^{*},{\beta }^{*})$, then

$$F({\mathit{r}}^{*},{\beta }^{*})\le F({\mathit{r}}^n,{\beta }^n)\le F({\mathit{q}}^n,{\tilde{\beta }}^n)\to F({\mathit{r}}^{*},{\beta }^{*}).$$

(A5)

Therefore, we obtain the convergence $F({\mathit{r}}^n,{\beta }^n)\to F({\mathit{r}}^{*},{\beta }^{*})$. In summary, we only need to prove $F({\mathit{q}}^n,{\tilde{\beta }}^n)\to F({\mathit{r}}^{*},{\beta }^{*})$. □

Appendix A.3. Proof of Lemma 2

Proof. 

(1). We first show that the sequence $\left\{{\tilde{\beta }}^n\right\}$ is bounded away from zero, i.e.,

$${\tilde{\beta }}^n\ge B_1>0,\forall n.$$

Suppose, to the contrary, that there exists a subsequence with ${\tilde{\beta }}^{n_k}\to 0$. Therefore, there exists $k_0$ such that for all $k\ge k_0$, ${\tilde{\beta }}^{n_k}<{\beta }^{*}/2$. Here we assume ${\beta }^{*}>0$ and the degenerate case ${\beta }^{*}=0$ can be disregarded. By optimality, ${\tilde{\beta }}^n$ satisfies the first-order optimality condition and thus is a root of a monotone function

$$G_{{\mathit{q}}^n}\left(\beta \right)=\int {\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^n\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^n\left(y\right)\right)}}\mathrm{d}p\left(x\right)-D.$$

Then by the monotone property of $G_{{\mathit{q}}^{n_k}}$, we have

$$0=G_{{\mathit{q}}^{n_k}}\left({\tilde{\beta }}^{n_k}\right)>G_{{\mathit{q}}^{n_k}}({\beta }^{*}/2).$$

The monotonicity follows from $G_{{\mathit{q}}^{n_k}}^{\prime }\left(\beta \right)<0$, which is ensured by the Cauchy–Schwarz inequality. Thus, for sufficiently large M,

$$\begin{array}{cc}\hfill D& >\int {\displaystyle \frac{e^{-{\beta }^{*}\rho (x,y)/2}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)}{\int e^{-{\beta }^{*}\rho (x,y)/2}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)}}\mathrm{d}p\left(x\right)\ge {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\int e^{-{\beta }^{*}\rho (x,y)/2}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)}{\int e^{-{\beta }^{*}\rho (x,y)/2}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)}}\mathrm{d}p\left(x\right).\hfill \end{array}$$

By the proof of Lemma 1, we have

$$\int e^{-{\beta }^{*}\rho (x,y)/2}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\rightrightarrows \int e^{-{\beta }^{*}\rho (x,y)/2}\mathrm{d}{\mathit{r}}^{*}\left(y\right).$$

Similarly,

$$\int e^{-{\beta }^{*}\rho (x,y)/2}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\rightrightarrows \int e^{-{\beta }^{*}\rho (x,y)/2}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right).$$

Similarly to (A2) in the proof of Lemma 1, we have

$$\int e^{-{\beta }^{*}\rho (x,y)/2}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\ge {\delta }_0.$$

Meanwhile, the upper bound can be estimated by

$$\int e^{-{\beta }^{*}\rho (x,y)/2}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\le \int B\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\le B.$$

Here, B is a bound of the function $e^{-{\beta }^{*}x/2}x,x\ge 0$.

Next, since $g(x,y)=x/y$ is uniformly continuous over $[0,B]\times [{\delta }_0,\infty )$, we have

$${\displaystyle \frac{\int \exp \left(-{\beta }^{*}\rho (x,y)/2\right)\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)}{\int \exp \left(-{\beta }^{*}\rho (x,y)/2\right)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)}}\rightrightarrows {\displaystyle \frac{\int \exp \left(-{\beta }^{*}\rho (x,y)/2\right)\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)}{\int \exp \left(-{\beta }^{*}\rho (x,y)/2\right)\mathrm{d}{\mathit{r}}^{*}\left(y\right)}}.$$

Thus, it holds that

$$\begin{array}{cc}\hfill D& \ge {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\int e^{-{\beta }^{*}\rho (x,y)/2}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)}{\int e^{-{\beta }^{*}\rho (x,y)/2}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)}}\mathrm{d}p\left(x\right)\to {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\int e^{-{\beta }^{*}\rho (x,y)/2}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)}{\int e^{-{\beta }^{*}\rho (x,y)/2}\mathrm{d}{\mathit{r}}^{*}\left(y\right)}}\mathrm{d}p\left(x\right).\hfill \end{array}$$

(A6)

Letting $M\to \infty$ leads to

$$G_{{\mathit{r}}^{*}}({\beta }^{*}/2)\le 0=G_{{\mathit{r}}^{*}}\left({\beta }^{*}\right).$$

Using the monotonicity of $G_{{\mathit{r}}^{*}}$, we have ${\beta }^{*}/2\ge {\beta }^{*}$, which is a contradiction.

(2). To prove ${\tilde{\beta }}^n\to {\beta }^{*}$, we only need to show that every convergent subsequence converges to ${\beta }^{*}$, i.e., if a convergent subsequence ${\tilde{\beta }}^{n_k}\to \overline{\beta }$, then $\overline{\beta }={\beta }^{*}$. Now, let ${\tilde{\beta }}^{n_k}$ be a convergent subsequence and ${\tilde{\beta }}^{n_k}\to \overline{\beta }$. Since ${\tilde{\beta }}^{n_k}$ is lower bounded away from zero, as shown in Lemma 2, we have $\overline{\beta }>0$. Let $A_0$ satisfy $e^{-\overline{\beta }t/2}t\le \epsilon$ and $e^{-\overline{\beta }t/2}\le \epsilon ,\forall t\ge A_0$. We have

$$\begin{array}{cc}\hfill & |\int e^{-{\tilde{\beta }}^{n_k}\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)-\int e^{-\overline{\beta }\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)|\hfill \\ \hfill & \le \int \left|e^{-{\tilde{\beta }}^{n_k}\rho (x,y)}-e^{-\overline{\beta }\rho (x,y)}\right|\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\le \int e^{-\overline{\beta }\rho (x,y)}|e^{(\overline{\beta }-{\tilde{\beta }}^{n_k})\rho (x,y)}-1|\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right).\hfill \end{array}$$

Next, we divide the integral into two parts and estimate each of them. On one hand, when k is sufficiently large,

$$\begin{array}{cc}\hfill & {\int }_{y:\rho (x,y)>A_0}{e}^{-\overline{\beta }\rho (x,y)}|e^{(\overline{\beta }-{\tilde{\beta }}^{n_k})\rho (x,y)}-1|\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le {\int }_{y:\rho (x,y)>A_0}{e}^{-\overline{\beta }\rho (x,y)}{e}^{|\overline{\beta }-{\tilde{\beta }}^{n_k}|\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le {\int }_{y:\rho (x,y)>A_0}{e}^{-\overline{\beta }\rho (x,y)/2}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\le {\int }_{y:\rho (x,y)>A_0}\epsilon \mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\le \epsilon ,\hfill \end{array}$$

where the second inequality is due to $|\overline{\beta }-{\tilde{\beta }}^{n_k}|\le \overline{\beta }/2$ when k is sufficiently large; on the other hand, when k is sufficiently large,

$$\begin{array}{cc}\hfill & {\int }_{y:\rho (x,y)\le A_0}{e}^{-\overline{\beta }\rho (x,y)}|e^{(\overline{\beta }-{\tilde{\beta }}^{n_k})\rho (x,y)}-1|\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le {\int }_{y:\rho (x,y)\le A_0}|e^{(\overline{\beta }-{\tilde{\beta }}^{n_k})\rho (x,y)}-1|A_0\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le {\int }_{y:\rho (x,y)\le A_0}(e^{|\overline{\beta }-{\tilde{\beta }}^{n_k}|\rho (x,y)}-1)A_0\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le {\int }_{y:\rho (x,y)\le A_0}(e^{|\overline{\beta }-{\tilde{\beta }}^{n_k}|A_0}-1)A_0\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le (e^{|\overline{\beta }-{\tilde{\beta }}^{n_k}|A_0}-1)A_0\le \epsilon .\hfill \end{array}$$

Combining the two parts, we obtain

$$|\int e^{-{\tilde{\beta }}^{n_k}\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)-\int e^{-\overline{\beta }\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)|\le 2\epsilon ,$$

when k is sufficiently large. Thus,

$$\int e^{-{\tilde{\beta }}^{n_k}\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)-\int e^{-\overline{\beta }\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\rightrightarrows 0,\forall x.$$

(A7)

By the proof of Lemma 2,

$$\begin{array}{c}\hfill \int e^{-\overline{\beta }\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\rightrightarrows \int e^{-\overline{\beta }\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right),\forall x.\end{array}$$

Thus,

$$\begin{array}{c}\hfill \int e^{-{\tilde{\beta }}^{n_k}\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\rightrightarrows \int e^{-\overline{\beta }\rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right),\forall x.\end{array}$$

Similarly,

$$\begin{array}{c}\hfill \int e^{-{\tilde{\beta }}^{n_k}\rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\rightrightarrows \int e^{-\overline{\beta }\rho (x,y)}\mathrm{d}{\mathit{r}}^{*}\left(y\right),\forall x.\end{array}$$

Next, we prove that $G_{{\mathit{q}}^{n_k}}\left({\tilde{\beta }}^{n_k}\right)\to G_{{\mathit{r}}^{*}}\left(\overline{\beta }\right)$. Since ${\tilde{\beta }}^{n_k}\to \overline{\beta }$, for any $\epsilon >0$ there exists $k_0$ such that for all $k\ge k_0$, $|{\tilde{\beta }}^{n_k}-\overline{\beta }|\le \epsilon$. Using the monotonicity of $G_{{\mathit{q}}^{n_k}}$, we obtain

$$G_{{\mathit{q}}^{n_k}}(\overline{\beta }+\epsilon )\le G_{{\mathit{q}}^{n_k}}\left({\tilde{\beta }}^{n_k}\right)\le G_{{\mathit{q}}^{n_k}}(\overline{\beta }-\epsilon ).$$

(A8)

In the following, we divide $G_{{\mathit{q}}^{n_k}}(\overline{\beta }+\epsilon )$ into two parts and estimate each of them. Set $\beta :=\overline{\beta }+\epsilon$ for brevity. We first estimate the part

$$\begin{array}{cc}\hfill & {\int }_{\parallel x\parallel >M}{\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}}\mathrm{d}p\left(x\right).\hfill \end{array}$$

Denote $A_x=-{\displaystyle \frac{1}{\beta }}\log \int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)$, then

$$\begin{array}{cc}\hfill & \int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le {\int }_{y:\rho (x,y)\le A_x}{e}^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)+{\int }_{y:\rho (x,y)>A_x}{e}^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le {\int }_{y:\rho (x,y)\le A_x}{e}^{-\beta \rho (x,y)}{A}_x\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)+{\int }_{y:\rho (x,y)>A_x}{e}^{-\beta \max (A_x,1/\beta )}\max (A_x,1/\beta )\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le A_x\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)+e^{-\beta \max (A_x,1/\beta )}\max (A_x,1/\beta )\hfill \\ \hfill & =A_x\exp (-\beta A_x)+e^{-\beta \max (A_x,1/\beta )}\max (A_x,1/\beta ).\hfill \end{array}$$

Thus,

$$\begin{array}{cc}\hfill & {\int }_{\parallel x\parallel >M}{\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}}\mathrm{d}p\left(x\right)\le {\int }_{\parallel x\parallel >M}\left(A_x+\max (A_x,1/\beta )\right)\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel >M}\left(2A_x+1/\beta \right)\mathrm{d}p\left(x\right).\hfill \end{array}$$

Note that when k is sufficiently large, we have

$$\begin{array}{cc}\hfill A_x& =-{\displaystyle \frac{1}{\beta }}\log \int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\le -{\displaystyle \frac{1}{\beta }}\log {\int }_{\parallel y\parallel \le A}{e}^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\hfill \\ \hfill & \le -{\displaystyle \frac{1}{\beta }}\log {\int }_{\parallel y\parallel \le A}{e}^{-\beta g\left(x\right)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\le -{\displaystyle \frac{1}{\beta }}\log (e^{-\beta g\left(x\right)}/2)=g\left(x\right)+\left(\log 2\right)/\beta ,\hfill \end{array}$$

Here, A is the constant in Theorem 1 and $g\left(x\right)={\max }_{\parallel y\parallel \le A}\rho (x,y)$. Then,

$$\int (2A_x+1/\beta )\mathrm{d}p\left(x\right)<\infty .$$

Similarly to (A6), the other part satisfies

$$\begin{array}{cc}\hfill & {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}}\mathrm{d}p\left(x\right)\hfill \\ \hfill & \to {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}}\mathrm{d}p\left(x\right),\mathrm{as}k\to \infty .\hfill \end{array}$$

Finally, combining the two parts, we have

$$\begin{array}{cc}\hfill & G_{{\mathit{q}}^{n_k}}\left(\beta \right)=\int {\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}}\mathrm{d}p\left(x\right)-D\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}}\mathrm{d}p\left(x\right)+{\int }_{\parallel x\parallel >M}(2A_x+1/\beta )\mathrm{d}p\left(x\right)-D.\hfill \end{array}$$

Letting $k\to \infty$ first,

$$\begin{array}{cc}\hfill & \underset{k}{\lim \sup }{G}_{{\mathit{q}}^{n_k}}\left(\beta \right)\le {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}}\mathrm{d}p\left(x\right)\hfill \\ \hfill & +{\int }_{\parallel x\parallel >M}(2A_x+1/\beta )\mathrm{d}p\left(x\right)-D.\hfill \end{array}$$

Then letting $M\to \infty$,

$$\begin{array}{cc}\hfill & \underset{k}{\lim \sup }{G}_{{\mathit{q}}^{n_k}}\left(\beta \right)\le \int {\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}}\mathrm{d}p\left(x\right)-D=G_{{\mathit{r}}^{*}}\left(\beta \right),\hfill \end{array}$$

i.e., ${\lim \sup }_k{G}_{{\mathit{q}}^{n_k}}(\overline{\beta }+\epsilon )\le G_{{\mathit{r}}^{*}}(\overline{\beta }+\epsilon )$. Meanwhile,

$$\begin{array}{cc}\hfill & G_{{\mathit{q}}^{n_k}}\left(\beta \right)=\int {\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}}\mathrm{d}p\left(x\right)-D\hfill \\ \hfill & \ge {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{q}}^{n_k}\left(y\right)\right)}}\mathrm{d}p\left(x\right)-D.\hfill \end{array}$$

Letting $k\to \infty$ and then $M\to \infty$, we have

$$\begin{array}{cc}\hfill & \underset{k}{\mathrm{lim\; inf}}{G}_{{\mathit{q}}^{n_k}}\left(\beta \right)\ge \underset{M\to \infty }{\lim }{\int }_{\parallel x\parallel \le M}{\displaystyle \frac{\left(\int e^{-\beta \rho (x,y)}\rho (x,y)\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}{\left(\int e^{-\beta \rho (x,y)}\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)}}\mathrm{d}p\left(x\right)-D=G_{{\mathit{r}}^{*}}\left(\beta \right),\hfill \end{array}$$

i.e., $\underset{k}{\mathrm{lim\; inf}}{G}_{{\mathit{q}}^{n_k}}(\overline{\beta }+\epsilon )\ge G_{{\mathit{r}}^{*}}(\overline{\beta }+\epsilon )$. Thus, we obtain ${\lim }_k{G}_{{\mathit{q}}^{n_k}}(\overline{\beta }+\epsilon )=G_{{\mathit{r}}^{*}}(\overline{\beta }+\epsilon )$. Similarly, ${\lim }_k{G}_{{\mathit{q}}^{n_k}}(\overline{\beta }-\epsilon )=G_{{\mathit{r}}^{*}}(\overline{\beta }-\epsilon )$. Using (A8),

$$\begin{array}{cc}\hfill G_{{\mathit{r}}^{*}}(\overline{\beta }+\epsilon )& =\underset{k}{\lim }{G}_{{\mathit{q}}^{n_k}}(\overline{\beta }+\epsilon )\le \underset{k}{\mathrm{lim\; inf}}{G}_{{\mathit{q}}^{n_k}}\left({\tilde{\beta }}^{n_k}\right)\hfill \\ & \le \underset{k}{\lim \sup }{G}_{{\mathit{q}}^{n_k}}\left({\tilde{\beta }}^{n_k}\right)\le \underset{k}{\lim }{G}_{{\mathit{q}}^{n_k}}(\overline{\beta }-\epsilon )=G_{{\mathit{r}}^{*}}(\overline{\beta }-\epsilon ).\hfill \end{array}$$

Then, let $\epsilon \to 0$, $0={\lim }_k{G}_{{\mathit{q}}^{n_k}}\left({\tilde{\beta }}^{n_k}\right)=G_{{\mathit{r}}^{*}}\left(\overline{\beta }\right)$. Therefore, $G_{{\mathit{r}}^{*}}\left(\overline{\beta }\right)=0=G_{{\mathit{r}}^{*}}\left({\beta }^{*}\right)$. By the strict monotonicity, we have $\overline{\beta }={\beta }^{*}$. Thus ${\tilde{\beta }}^n\to {\beta }^{*}$ holds. □

Appendix A.4. Proof of Lemma 3

Proof. 

By [33] (Lemma 4.1), the distribution $\mathit{r}$ of the reproduction variable is supported on ${[-M,M]}^d$ whenever $\rho (x,y)$ is a strictly increasing, continuous difference distortion measure. We can just take the equidistant discretization nodes ${\left\{y_j^n\right\}}_{j=1}^n$ from ${[-M,M]}^d$. When $x,y\in {[-M,M]}^d$, $\exp (-\beta \rho (x,y\left)\right)$ is a Lipschitz continuous function with respect to y with L being the Lipschitz constant. Then, we have the evaluation

$$\begin{array}{cc}\hfill & |\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)-\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)|\hfill \\ \hfill & \le \sum _i{\int }_{I_i}|\exp (-\beta \rho (x,y))-\exp (-\beta \rho (x,y_i^n))|\mathrm{d}{\mathit{r}}^{*}\left(y\right)\le \sum _i{\int }_{I_i}L\parallel y-y_i^n\parallel \mathrm{d}{\mathit{r}}^{*}\left(y\right)\le Lh/2.\hfill \end{array}$$

We now establish the convergence rate:

$$\begin{array}{cc}\hfill f\left({\mathit{q}}^n\right)-f\left({\mathit{r}}^{*}\right)& ={\int }_{\parallel x\parallel \le M}[-\log \left(\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)\right)\hfill \\ \hfill & +\log \left(\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)\right)]\mathrm{d}p\left(x\right)\hfill \\ \hfill & ={\int }_{\parallel x\parallel \le M}\log {\displaystyle \frac{\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{r}}^{*}\left(y\right)}{\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}\mathrm{d}p\left(x\right)\hfill \\ \hfill & ={\int }_{\parallel x\parallel \le M}\log \left({\displaystyle \frac{{\tilde{\mathcal{H}}}_{{\mathit{r}}^{*}\left(y\right)}\left(x\right)-{\tilde{\mathcal{H}}}_{{\mathit{q}}^n\left(y\right)}\left(x\right)}{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}+1\right)\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{|{\tilde{\mathcal{H}}}_{{\mathit{r}}^{*}\left(y\right)}\left(x\right)-{\tilde{\mathcal{H}}}_{{\mathit{q}}^n\left(y\right)}\left(x\right)|}{\int \exp (-{\tilde{\beta }}^n\rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)}}\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}{\displaystyle \frac{|{\tilde{\mathcal{H}}}_{{\mathit{r}}^{*}\left(y\right)}\left(x\right)-{\tilde{\mathcal{H}}}_{{\mathit{q}}^n\left(y\right)}\left(x\right)|}{{\delta }_0}}\mathrm{d}p\left(x\right)\hfill \\ \hfill & \le {\int }_{\parallel x\parallel \le M}Lh/\left(2\delta \right)\mathrm{d}p\left(x\right)\le Lh/\left(2\delta \right)=O\left(h\right),\hfill \end{array}$$

where $\delta$ is the lower bound of $\int \exp (-\beta \rho (x,y))\mathrm{d}{\mathit{q}}^n\left(y\right)$ as shown in Theorem 1 and ${\tilde{\mathcal{H}}}_{\mathit{r}\left(y\right)}\left(x\right)=\int \exp (-\beta \rho (x,y))\mathrm{d}\mathit{r}\left(y\right)$. Thus,

$$f\left({\mathit{r}}^n\right)-f\left({\mathit{r}}^{*}\right)=O\left(h\right).$$

Since the total number of discrete nodes is n, there are $n^{{\displaystyle \frac{1}{d}}}$ nodes in each dimension. So, the discretization step size is ${\displaystyle \frac{2M}{n^{1/d}}}$. Therefore, the convergence rate is $O(1/n^{{\displaystyle \frac{1}{d}}})$. □