Soft Quantization Using Entropic Regularization

Abstract

The quantization problem aims to find the best possible approximation of probability measures on ${\mathbb{R}}^d$ using finite and discrete measures. The Wasserstein distance is a typical choice to measure the quality of the approximation. This contribution investigates the properties and robustness of the entropy-regularized quantization problem, which relaxes the standard quantization problem. The proposed approximation technique naturally adopts the softmin function, which is well known for its robustness from both theoretical and practicability standpoints. Moreover, we use the entropy-regularized Wasserstein distance to evaluate the quality of the soft quantization problem’s approximation, and we implement a stochastic gradient approach to achieve the optimal solutions. The control parameter in our proposed method allows for the adjustment of the optimization problem’s difficulty level, providing significant advantages when dealing with exceptionally challenging problems of interest. As well, this contribution empirically illustrates the performance of the method in various expositions.

1. Introduction

Over the past few decades, extensive research has been conducted on optimal quantization techniques in order to tackle numerical problems that are related to various fields such as data science, applied disciplines, and economic models. These problems are typically centered around uncertainties or probabilities which demand robust and efficient solutions (cf. Graf and Mauldin [1], Luschgy and Pagès [2], El Nmeir et al. [3]). In general, these problems are difficult to handle, as the random components in the problem allow uncountably many outcomes. As a consequence, in order to address this difficulty the probability measures are replaced by simpler or finite measures, which can facilitate numerical computations. However, the probability measures should be ‘close’ in order to ensure that the result of the computations with approximate (discrete) measures resembles the original problem. In a nutshell, the goal is to find the best approximation of a diffuse measure using a discrete measure, which is called an optimal quantization problem. For a comprehensive discussion of the optimal quantization problem from a mathematical standpoint, refer to Graf and Luschgy [4].

On the other hand, entropy (sometimes known as information entropy) is an essential concept when dealing with uncertainties and probabilities. In mathematics, entropy is often used as a measure of information and uncertainty. It provides a quantitative measure of the randomness or disorder in a system or a random variable. Its applications span information theory, statistical analysis, probability theory, and the study of complex dynamical systems (cf. Breuer and Csiszár [5,6], Pichler and Schlotter [7]).

In order to assess the closeness of probability measures, distances are often considered; one notable instance is the Wasserstein distance. Ostensibly, the Wasserstein distance measures the minimum average amount of transporting cost required to transfer one probability measure into another. Unlike other formulations of distances and/or divergence, which simply compare the probabilities of the distribution functions (e.g., the total variation distance and the Kullback–Leibler divergence), the Wasserstein distance incorporates the geometry of the underlying space. This increases the understanding of the relationships between different probability measures in a geometrically trustworthy manner.

In our research work, we focus on entropy-regularized quantization methods. More precisely, we consider an entropy-regularized version of the Wasserstein problem to quantify the quality of the approximation, and adapt the stochastic gradient approach to obtain the optimal quantizers.

The key features of our methodology include the following:

(i)

Our regularization approach stabilizes and simplifies the standard quantization problem by introducing penalty terms or constraints that discourage overly complex or overfitted models, promoting better generalizations and robustness in the solutions.

(ii)

The influence of entropy is controlled using a parameter, $\lambda$, which enables us to reach the genuine optimal quantizers.

(iii)

Generally, parameter tuning comes with certain limitations. However, our method builds upon the framework of the well-established softmin function, which allows us to exercise parameter control without encountering any restrictions.

(iv)

For larger values of the regularization parameter $\lambda$, the optimal measure accumulates all its mass at the center of the measure.

• Applications in the Context of Quantization.

Quantization techniques have undergone significant developments in recent years, particularly within the domain of deep learning and model optimization. State-of-the-art research has introduced advanced methodologies such as non-uniform quantization and quantization-aware training, enabling the efficient deployment of neural networks while preserving performance (cf. Jacob et al. [8], Zhuang et al. [9], Hubara et al. [10]). Furthermore, quantization principles have found applications beyond machine learning, such as in digital image processing, computer vision (cf. Polino et al. [11]), and electric charge quantization (cf. Bhattacharya [12]).

• Related Works and Contributions.

As mentioned above, optimal quantization is a well-researched topic in the field of information theory and signal processing. Several methods have been developed for the optimal quantization problem, notably including the following:

–

Lloyd-Max Algorithm: this algorithm, also known as Lloyd’s algorithm or the k-means algorithm, is a popular iterative algorithm for computing optimal vector quantizers. It iteratively adjusts the centroids of the quantization levels to minimize the quantization error (cf. Scheunders [13]).

–

Tree-Structured Vector Quantization (TSVQ): TSVQ is a hierarchical quantization method that uses a tree structure to partition the input space into regions. It recursively applies vector quantization at each level of the tree until the desired number of quantization levels is achieved (cf. Wei and Levoy [14]).

–

Expectation-maximization (EM) algorithm: the EM algorithm is a general-purpose optimization algorithm that can be used for optimal quantization. It is an iterative algorithm that estimates the parameters of a statistical model to maximize the likelihood of the observed data (cf. Heskes [15]).

–

Stochastic Optimization Methods: stochastic optimization methods such as simulated annealing, genetic algorithms, and particle swarm optimization can be used to find optimal quantization strategies by exploring the search space and iteratively improving the quantization performance (cf. Pagès et al. [16]).

–

Greedy vector quantization (GVQ): the greedy algorithm tries to solve this problem iteratively by adding one code word at every step until the desired number of code words is reached, each time selecting the code word that minimizes the error. GVQ is known to provide suboptimal quantization compared to other non-greedy methods such as the Lloyd-Max and Linde–Buzo–Gray algorithms. However, it has been shown to perform well when the data have a strong correlation structure. Notably, it utilizes the Wasserstein distance to measure the error of approximation (cf. Luschgy and Pagès [2]).

These methods provide efficient and practical solutions for finding optimal quantization schemes, and have different trade-offs between complexity and performance. The choice of method depends on the problem of interest and the requirements of the application. However, most of these methods depend on strict constraints, which makes the solutions overly complex or results in model overfitting. Our method mitigates this issue by promoting better generalizations and robustness in the solutions.

In the optimal transport community, the entropy-regularized version of the optimal transport problem (known as the entropy-regularized Wasserstein problem) was initial proposed in Cuturi [17]. This entropy version of the Wasserstein problem promotes fast computations using Sinkhorn’s algorithm. As an avenue for constructive research, the above-cited study presented a multitude of results aimed at gaining a comprehensive understanding of the subtleties involved in enhancing the computational performance of entropy-optimal transport (cf. Ramdas et al. [18], Neumayer and Steidl [19], Altschuler et al. [20], Lakshmanan et al. [21], Ba and Quellmalz [22], Lakshmanan and Pichler [23]). These findings have served as a valuable foundation for further exploration in the field of optimal transport, providing insights into both the intricacies of the topic and potential avenues for improvement.

In contrast, we present a new and innovative approach that concentrates on the optimal quantization problem based on entropy and on its robust properties, which represents a distinct contribution with regard to standard entropy-regularized optimal transport problems.

One of the principal consequences of our research substantiates the convergence behavior of quantizers at the center of the measure. The relationship between the center of the measure and the entropy-regularized quantization problem has not been exposed yet. The following plain solution is obtained by intensifying the entropy term in the regularization of the quantization problem.

Theorem 1.

There exists a real valued ${\lambda }_0>0$ such that the approximation of the entropy-regularized optimal quantization problem is provided by the Dirac measure

$$P={\delta }_a$$

for every $\lambda >{\lambda }_0$, where a is the center of the measure P with respect to the distance d.

This enthralling interpretation (Theorem 1) of our master problem facilitates an understanding of the transition from a complex and difficult optimization solution to a simple solution. Moreover, along with a theoretical discussion, we provide an algorithm and numerical exemplification which empirically demonstrate the robustness of our method. The forthcoming sections elucidate the robustness and asymptotic properties of the proposed method in detail.

• Outline of the Paper.

Section 2 establishes the essential notation, definitions, and properties. Moreover, we comprehensively expound upon the significance of the smooth minimum, a pivotal component in our research. In Section 3, we introduce the entropy-regularized optimal quantization problem and delve into its inherent properties. Section 4 presents a discussion of the soft tessellation, optimal weights, and theoretical properties of parameter tuning. Furthermore, we systematically illustrate the computational process along with a pseudo-algorithm. Section 5 provides numerical examples and empirically substantiates the theoretical proofs. Finally, Section 6 summarizes the study.

2. Preliminaries

In what follows, $(\mathcal{X},d)$ is a Polish space. The $\sigma$-algebra generated by the Borel sets induced by the distance d is $\mathcal{F}$, while the set of all probability measures on $\mathcal{X}$ is $\mathcal{P}\left(\mathcal{X}\right)$.

2.1. Distances and Divergences of Measures

The standard quantization problem employs the Wasserstein distance to measure the quality of the approximation, which was initially studied by Monge and Kantorovich (cf. Monge [24], Kantorovich [25]). One of the remarkable properties of this distance is that it metrizes the weak* topology of measures.

Definition 1

(Wasserstein distance). Let P and $\tilde{P}$ be probability measures on $(\mathcal{X},d)$. The Wasserstein distance of order $r\ge 1$ between P and $\tilde{P}\in \mathcal{P}\left(\mathcal{X}\right)$ is

$$d_r(P,\tilde{P}):=inf{\left(\underset{\mathcal{X}\times \mathcal{X}}{\int \int }d{(\xi ,\tilde{\xi })}^r \pi (\mathrm{d}\xi ,\mathrm{d}\tilde{\xi })\right)}^{1/r},$$

(1)

where the infimum is among all measures $\pi \in \mathcal{P}\left({\mathcal{X}}^2\right)$ with marginals P and $\tilde{P}$, that is,

$$\begin{array}{cc}\hfill \pi (A\times \mathcal{X})& =P\left(A\right)and\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill \pi (\mathcal{X}\times B)& =\tilde{P}\left(B\right)\hfill \end{array}$$

(3)

for all sets A and $B\in \mathcal{F}$. The measures

$${\pi }_1(·):=\pi (·\times \mathcal{X}) and {\pi }_2(·):=\pi (\mathcal{X}\times ·)$$

on $\mathcal{X}$ are called the marginal measures of the bivariate measure π.

Readers may refer to the excellent monographs in [26,27] for a comprehensive discussion of the Wasserstein distance.

Remark 1

(Flexibility). In the subsequent discussion, our problem of interest is to approximate the measure P, which is a continuous, discrete, or mixed measure on $\mathcal{X}={\mathbb{R}}^d$. The measure $\tilde{P}$ is used to approximate the measure P, which is a discrete measure. The definition of the Wasserstein distance flexibly comprises all the cases, namely, continuous, semi-discrete, and discrete measures.

In contrast to the standard methodology, we investigate the quantization problem by utilizing an entropy version of the Wasserstein distance. The standard Wasserstein problem is regularized by adding the Kullback–Leibler divergence, which is known as the relative entropy.

Definition 2

(Kullback–Leibler divergence). Let P and $Q\in \mathcal{P}\left(\mathcal{X}\right)$ be probability measures. Denote the Radon–Nikodým derivative $\mathrm{d}Q=Z \mathrm{d}P$ by $Z\in L^1\left(P\right)$ if Q is absolutely continuous with respect to P ($Q\ll P$). The Kullback–Leibler divergence is

$$D(Q\parallel P):=\left\{\begin{array}{cc}{\mathbb{E}}_{P}ZlogZ={\mathbb{E}}_{Q}logZ\hfill & ifQ\ll Pand\mathrm{d}Q=Z \mathrm{d}P,\hfill \\ +\infty \hfill & else,\hfill \end{array}\right.$$

(4)

where ${\mathbb{E}}_P$ (${\mathbb{E}}_Q$, resp.) is the expectation with respect to the measure P (Q, resp.).

Per Gibbs’ inequality, the Kullback–Leibler divergence satisfies $D(Q\parallel P)\ge 0$ (non-negativity). However, D is not a distance metric, as it does not satisfy the symmetry and triangle inequality properties.

We would like to emphasize the following distinctness with respect to the Wasserstein distance (cf. Remark 1): in order for the Kullback–Leibler divergence to be finite ($D(Q\parallel P)<\infty$), we must have

$$\mathrm{supp}Q\subset \mathrm{supp}P,$$

where the support of the measure (cf. Rüschendorf [28]) is

$$\mathrm{supp}P:=\bigcap \left\{A\in \mathcal{F}:A\mathrm{is}\mathrm{closed}\mathrm{and}P\left(A\right)=1\right\}.$$

If P is a continuous measure on $\mathcal{X}={\mathbb{R}}^d$, then Q is as well. If P is a finite measure, then the support points of P contain the support points of Q.

2.2. The Smooth Minimum

In what follows, we present the smooth minimum in its general form, which includes discrete and continuous measures. The numerical computations in the following section rely on results for its discrete version. Therefore, we address the special properties of its discrete version in detail.

Definition 3

(Smooth minimum). Let $\lambda >0$ and let Y be a random variable. The smooth minimum, or smooth minimum with respect to $\tilde{P}$, is

$$\begin{array}{cc}\hfill {min}_{\tilde{P}; \lambda }\left(Y\right)& :=-\lambda log{\mathbb{E}}_{\tilde{P}}{e}^{-Y/\lambda }\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & =-\lambda log{\int }_{\mathcal{X}}{e}^{-Y\left(\eta \right)/\lambda } \tilde{P}\left(\mathrm{d}\eta \right),\hfill \end{array}$$

(6)

provided that the expectation (integral) of $e^{-Y/\lambda }$ is finite, or if it is not finite, that ${min}_{\tilde{P}; \lambda }\left(Y\right):=-\infty$. For $\lambda =0$, we set

$${min}_{\tilde{P}; \lambda =0}\left(Y\right):=ess\; infY.$$

(7)

For a σ-algebra $\mathcal{G}\subset \mathcal{F}$ and $\lambda >0$ measurable with respect to $\mathcal{G}$, the conditional smooth minimum is

$${min}_{\tilde{P};\lambda }\left(Y\right| \mathcal{G}):=-\lambda log{\mathbb{E}}_{\tilde{P}}\left(\left.e^{-Y/\lambda }\right| \mathcal{G}\right).$$

The following lemma relates the smooth minimum with the essential infimum (cf. (7)), that is, colloquially, the ‘minimum’ of a random variable. As well, the result justifies the term smooth minimum.

Lemma 1.

For $\lambda >0$, it holds that

$${min}_{\tilde{P};\lambda }\left(Y\right)\le {\mathbb{E}}_{\tilde{P}}Y$$

(8)

and

$$ess\; infY\le {min}_{\tilde{P};\lambda }\left(Y\right)\underset{\lambda \to 0}{\to }ess\; infY.$$

(9)

Proof. 

Inequality (8) follows from Jensen’s inequality as applied to the convex function $x\mapsto exp(-x/\lambda )$.

Next, the first inequality in the second display (9) follows from $ess\; infY\le Y$ and the fact that all operations in (6) are monotonic. Finally, let $a>ess\; infY$. Per Markov’s inequality, we have

$${\mathbb{E}}_{\tilde{P}}{e}^{-Y/\lambda }\ge e^{-a/\lambda } \tilde{P}\left(e^{-Y/\lambda }\ge e^{-a/\lambda }\right)=e^{-a/\lambda } \tilde{P}(Y\le a),$$

(10)

which is a variant of the Chernoff bound. From Inequality (10), it follows that

$${min}_{\tilde{P};\lambda }\left(Y\right)=-\lambda log{\mathbb{E}}_{\tilde{P}}{e}^{-Y/\lambda }\le -\lambda log\left(e^{-a/\lambda } \tilde{P}(Y\le a)\right)=a+\lambda log\frac{1}{\tilde{P}(Y\le a)}.$$

(11)

When $\lambda >0$ and $\lambda \to 0$, we have

$${min}_{\tilde{P};\lambda }\left(Y\right)\le a,$$

where a is an arbitrary number with $a>ess\; infY$. This completes the proof.    □

Remark 2

(Nesting property). The main properties of the smooth minimum include translation equivariance

$${min}_{\tilde{P};\lambda }(Y+c)={min}_{\tilde{P};\lambda }\left(Y\right)+c, c\in \mathbb{R},$$

and positive homogeneity

$${min}_{\tilde{P};\gamma ·\lambda }(\gamma ·Y)=\gamma ·{min}_{\tilde{P};\lambda }\left(Y\right), \gamma >0.$$

As a consequence of the tower property of the expectation, we have the nesting property

$${min}_{\tilde{P};\lambda }\left({min}_{\tilde{P};\lambda }\left(Y\right| \mathcal{G})\right)={min}_{\tilde{P};\lambda }\left(Y\right),$$

provided that $\mathcal{G}$ is a sub-σ-algebra of $\mathcal{F}$.

2.3. Softmin Function

The smooth minimum is related to the softmin function via its derivatives. In what follows, we express variants of its derivatives, which are involved later.

Definition 4

(Softmin function). For $\lambda >0$ and a random variable Y with a finite smooth minimum, the softmin function is the random variable

$${\sigma }_{\lambda }\left(Y\right):=exp\left(-\frac{Y-{min}_{\tilde{P};\lambda }\left(Y\right)}{\lambda }\right)=\frac{e^{-Y/\lambda }}{{\mathbb{E}}_{\tilde{P}}{e}^{-Y/\lambda }},$$

(12)

where the latter equality is obvious based on the definition of the smooth minimum in (6). The function ${\sigma }_{\lambda }\left(Y\right)$ is called the Gibbs density.

• The Derivative with respect to the Probability Measure

The definition of the smooth minimum in (6) does not require the measure $\tilde{P}$ to be a probability measure. Based on $\frac{\partial }{\partial t}log(a+t·h)=\frac{h}{a}$ (at $t=0$) for the natural logarithm, the directional derivative of the smooth minimum in the direction of the measure Q is

$$\begin{array}{c}\hfill \frac{1}{t}\left({min}_{\tilde{P}+t·Q;\lambda }\left(Y\right)-{min}_{\tilde{P};\lambda }\left(Y\right)\right)\end{array}$$

(13)

$$\begin{array}{cc}\hfill & =-\frac{\lambda }{t}\left(log{\int }_{\mathcal{X}}{e}^{-Y/\lambda } \mathrm{d}\left(\tilde{P}+t·Q\right)-log{\int }_{\mathcal{X}}{e}^{-Y/\lambda }\mathrm{d}\tilde{P}\right)\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & \underset{t\to 0}{\to }-\lambda ·\frac{{\int }_{\mathcal{X}}{e}^{-Y/\lambda } \mathrm{d}Q}{{\int }_{\mathcal{X}}{e}^{-Y/\lambda }\mathrm{d}\tilde{P}}\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & =-\lambda ·{\int }_{\mathcal{X}}{\sigma }_{\lambda }\left(Y\right) \mathrm{d}Q.\hfill \end{array}$$

(16)

Note that $-\lambda {\sigma }_{\lambda }$ is (up to the constant $-\lambda$) a Radon–Nikodým density in (16). Thus, the Gibbs density ${\sigma }_{\lambda }\left(Y\right)$ is proportional to the directional derivative of the smooth minimum with respect to the underlying measure $\tilde{P}$.

• The Derivative with respect to the Random Variable

In what follows, we additionally require the derivative of the smooth minimum with respect to its argument. Following similar reasoning as above, this is accomplished by

$$\begin{array}{c}\hfill \frac{1}{t}\left({min}_{\tilde{P};\lambda }(Y+t·Z)-{min}_{\tilde{P};\lambda }\left(Y\right)\right)\end{array}$$

(17)

$$\begin{array}{cc}\hfill & =-\frac{\lambda }{t}\left(log{\int }_{\mathcal{X}}{e}^{-(Y+t·Z)/\lambda } \mathrm{d}\tilde{P}-log{\int }_{\mathcal{X}}{e}^{-Y/\lambda }\mathrm{d}\tilde{P}\right)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & =-\frac{\lambda }{t}\left(log{\int }_{\mathcal{X}}{e}^{-Y/\lambda }\left(1-\frac{t}{\lambda }Z+𝒪\left(t^2\right)\right) \mathrm{d}\tilde{P}-log{\int }_{\mathcal{X}}{e}^{-Y/\lambda }\mathrm{d}\tilde{P}\right)\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & \underset{t\to 0}{\to }\frac{{\int }_{\mathcal{X}}Z·e^{-Y/\lambda } \mathrm{d}\tilde{P}}{{\int }_{\mathcal{X}}{e}^{-Y/\lambda }\mathrm{d}\tilde{P}}\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & ={\int }_{\mathcal{X}}Z·{\sigma }_{\lambda }\left(Y\right)\mathrm{d}\tilde{P},\hfill \end{array}$$

(21)

which involves the softmin function ${\sigma }_{\lambda }(·)$ as well.

3. Regularized Quantization

This section introduces the entropy-regularized optimal quantization problem along with its properties; we first recall the standard optimal quantization problem.

The standard quantization measures the quality of the approximation using the Wasserstein distance and considers the following problem (cf. Graf and Luschgy [4]):

$$\underset{\pi :\begin{array}{c}{\pi }_1=P, \\ {\pi }_2\in {\mathcal{P}}_m\left(\mathcal{X}\right)\end{array}}{inf}\underset{\mathcal{X}\times \mathcal{X}}{\int \int }d(\xi ,\tilde{\xi }) \pi (\mathrm{d}\xi ,\mathrm{d}\tilde{\xi }),$$

(22)

where

$${\mathcal{P}}_m\left(\mathcal{X}\right):=\left\{{\tilde{P}}_m\in \mathcal{P}\left(\mathcal{X}\right):{\tilde{P}}_m=\sum _{j=1}^m{\tilde{p}}_j {\delta }_{y_j}\right\}$$

(23)

is the set of measures on $\mathcal{X}$ supported by not more than m ($m\in \mathbb{N}$) points.

Soft quantization (or quantization regularized with the Kullback–Leibler divergence) involves the regularized Wasserstein distance instead of (22). The soft quantization problem is regularized with the Kullback–Leibler divergence:

$$inf\left\{{\mathbb{E}}_{\pi }{d}^r+\lambda ·D(\pi \parallel P\times {\tilde{P}}_m):{\pi }_1=P\mathrm{and}{\pi }_2={\tilde{P}}_m\in {\mathcal{P}}_m\left(\mathcal{X}\right)\right\},$$

(24)

where $\lambda >0$ and ${\mathbb{E}}_{\pi }{d}^r={\int \int }_{{\mathcal{X}}^2}d{(\xi ,\tilde{\xi })}^r \pi (\mathrm{d}\xi ,\mathrm{d}\tilde{\xi })$. The optimal measure ${\tilde{P}}_m\in {\mathcal{P}}_m\left(\mathcal{X}\right)$ for solving (24) depends on the regularization parameter $\lambda$.

In the following discussion, we initially investigate the regularized approximation, which again demonstrates the existence of an optimal approximation.

3.1. Approximation with Inflexible Marginal Measures

The following proposition addresses the optimal approximation problem after being regularized with the Kullback–Leibler divergence and fixed marginals. To this end, we dissect the infimum in the soft quantization problem (24) as follows:

$$\underset{{\tilde{P}}_m\in {\mathcal{P}}_m\left(\mathcal{X}\right)}{inf}\underset{\pi :\begin{array}{c}{\pi }_1=P,\\ {\pi }_2={\tilde{P}}_m\end{array}}{inf}{\mathbb{E}}_{\pi }{d}^r+\lambda ·D(\pi \parallel P\times {\tilde{P}}_m),$$

(25)

where the marginals P and ${\tilde{P}}_m$ are fixed in the inner infimum.

The following Proposition 1 addresses this problem with a fixed bivariate distribution, which is the inner infimum in (25). Then, Proposition 2 reveals that the optimal marginals coincide in this case.

Proposition 1.

Let P be a probability measure and let $\lambda >0$. The inner optimization problem in (25) relative to the fixed bivariate distribution $P\times \tilde{P}$ is provided by the explicit formula

$$\begin{array}{cc}\hfill \underset{\pi :{\pi }_1=P}{inf}{\mathbb{E}}_{\pi }{d}^r+\lambda ·D(\pi \parallel P\times \tilde{P})& =-\lambda {\int }_{\mathcal{X}}log{\int }_{\mathcal{X}}{e}^{-d{(\xi ,\tilde{\xi })}^r/\lambda } \tilde{P}\left(\mathrm{d}\tilde{\xi }\right) P\left(\mathrm{d}\xi \right)\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_{\xi \sim P}\left({min}_{\tilde{\xi }\sim \tilde{P};\lambda }d{(\xi ,\tilde{\xi })}^r\right),\hfill \end{array}$$

(27)

where $D(\pi \parallel P\times \tilde{P})$ is the Kullback–Leibler divergence. Further, the infimum in (26) is attained.

Remark 3.

The notation in (27) ((29) below, resp.) is chosen to reflect the explicit expression in (26), while the soft minimum ${min}_{\tilde{P};\lambda }$ is with respect to the measure $\tilde{P}$, which is associated with the variable $\tilde{\xi }$, and the expectation ${\mathbb{E}}_P$ is with respect to P and has an associated variable ξ (that is, the variable ξ in (27) is associated with P and the variable $\tilde{\xi }$ with $\tilde{P}$).

Remark 4

(Standard quantization). The result from (27) extends

$$\begin{array}{cc}\hfill \underset{\pi :\begin{array}{c}{\pi }_1=Pand\\ \mathrm{supp}{\pi }_2=\mathrm{supp}\tilde{P}\end{array}}{inf}& ={\int }_{\mathcal{X}}\underset{\xi \in \mathrm{supp}\tilde{P}}{min}d{(\xi ,\tilde{\xi })}^r P\left(\mathrm{d}\xi \right)\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_P\left(\underset{\tilde{\xi }\in \mathrm{supp}\tilde{P}}{min}d{(\xi ,\tilde{\xi })}^r\right),\hfill \end{array}$$

(29)

which is the formula without regularization and with restriction to the marginals P and $\tilde{P}$ (i.e., $\lambda =0$, cf. Pflug and Pichler [29]). Note that the preceding display thereby explicitly involves the support $\mathrm{supp}\tilde{P}$, while (26) only involves the expectation (via the smooth minimum) with respect to the measure $\tilde{P}$. In other words, (27) quantifies the quality of entropy-regularized quantization, while (29) quantifies standard quantization.

Proof of Proposition 1.

It follows from the definition of the Kullback–Leibler divergence in (4) that it is enough to consider measures $\pi$ which are absolutely continuous with respect to the product measure $\pi \ll P\times \tilde{P}$; otherwise, the objective is not finite. Hence, there is a Radon–Nikodým density $\tilde{Z}$ such that, with Fubini’s theorem,

$$\pi (A\times B)={\int }_A{\int }_B\tilde{Z}(\xi ,\eta ) \tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right).$$

In order for the marginal constraint $\pi (A\times \mathcal{X})=P\left(A\right)$ to be satisfied (cf. (2)), we have

$${\int }_A{\int }_{\mathcal{X}}\tilde{Z}(\xi ,\eta )\tilde{P}\left(\mathrm{d}\eta \right) P\left(\mathrm{d}\xi \right)=\pi (A\times \mathcal{X})=P\left(A\right)={\int }_{A}1 P\left(\mathrm{d}\xi \right)$$

for every measurable set A. It follows that

$${\int }_{\mathcal{X}}\tilde{Z}(\xi ,\eta ) \tilde{P}\left(\mathrm{d}\eta \right)=1 P\left(\mathrm{d}\xi \right)\mathrm{almost}\mathrm{everywhere}.$$

We can conclude that every density of the form

$$\tilde{Z}(\xi ,\eta )=\frac{Z(\xi ,\eta )}{{\int }_{\mathcal{X}}Z(\xi ,{\eta }^{\prime }) \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}$$

(30)

satisfies constraints in (2), irrespective of Z, and conversely that, via $\tilde{Z}$ in (30), every Z defines a bivariate measure $\pi$ satisfying the constraints in (2). We set $\Phi (\xi ,\eta ):=logZ(\xi ,\eta )$ (with the convention that $log0=-\infty$ and $exp(-\infty )=0$, resp.) and consider

$$\tilde{Z}(\xi ,\eta )=\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}.$$

With these, the divergence is

$$\begin{array}{cc}\hfill D(\pi \parallel P& \times \tilde{P})=\hfill \\ \hfill & ={\int }_{\mathcal{X}}{\int }_{\mathcal{X}}\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}log\frac{e^{\Phi (\xi ,\eta )}\tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right)}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)P\left(\mathrm{d}\xi \right)\tilde{P}\left(\mathrm{d}\eta \right)}\tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right)\hfill \\ \hfill & ={\int }_{\mathcal{X}}{\int }_{\mathcal{X}}\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}log\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}\tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right)\hfill \\ \hfill & ={\int }_{\mathcal{X}}{\int }_{\mathcal{X}}\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}\Phi (\xi ,\eta ) \tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right)\hfill \\ \hfill & -{\int }_{\mathcal{X}}\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}log{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)\tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right)\hfill \\ \hfill & ={\int }_{\mathcal{X}}{\int }_{\mathcal{X}}\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}\Phi (\xi ,\eta )-log{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)P\left(\mathrm{d}\xi \right).\hfill \end{array}$$

For the other term in Objective (24), we have

$${\mathbb{E}}_{\pi }{d}^r={\int }_{\mathcal{X}}{\int }_{\mathcal{X}}\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}d{(\xi ,\eta )}^r \tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right).$$

Combining the last expressions obtained, the objective in (26) is

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\pi }{d}^r+\lambda D(\pi \parallel P\times \tilde{P})& ={\int }_{\mathcal{X}}{\int }_{\mathcal{X}}\frac{e^{\Phi (\xi ,\eta )}}{{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}\left(d{(\xi ,\eta )}^r+\lambda \Phi (\xi ,\eta )\right) \tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right)\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill & -\lambda {\int }_{\mathcal{X}}log{\int }_{\mathcal{X}}{e}^{\Phi (\xi ,{\eta }^{\prime })} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)P\left(\mathrm{d}\xi \right).\hfill \end{array}$$

(32)

For fixed $\xi$ ($\xi$ is suppressed in the following two displays to abbreviate the notation), consider the function

$$f(\Phi ):={\int }_{\mathcal{X}}\frac{e^{\Phi \left(\eta \right)}}{{\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}\left(d{\left(\eta \right)}^r+\lambda \Phi \left(\eta \right)\right) \tilde{P}\left(\mathrm{d}\eta \right)-\lambda log{\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right).$$

The directional derivative in direction h of this function is

$$\begin{array}{c}\hfill \underset{t\to 0}{lim}\frac{1}{t}\left(f(\Phi +t h)-f(\Phi )\right)\end{array}$$

(33)

$$\begin{array}{cc}\hfill & ={\int }_{\mathcal{X}}\frac{e^{\Phi \left(\eta \right)}}{{\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}\left(d{\left(\eta \right)}^r+\lambda \Phi \left(\eta \right)-\lambda \right)h\left(\eta \right) \tilde{P}\left(\mathrm{d}\eta \right)\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill & -{\int }_{\mathcal{X}}\frac{e^{\Phi \left(\eta \right)}{\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)}h\left({\eta }^{\prime }\right) \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}{{\left({\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)\right)}^2}\left(d{\left(\eta \right)}^r+\lambda \Phi \left(\eta \right)\right) \tilde{P}\left(\mathrm{d}\eta \right)\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & +\lambda {\int }_{\mathcal{X}}\frac{e^{\Phi \left(\eta \right)}h\left(\eta \right)}{{\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}\tilde{P}\left(\mathrm{d}\eta \right)\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill & ={\int }_{\mathcal{X}}\frac{e^{\Phi \left(\eta \right)}}{{\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}\left(d{\left(\eta \right)}^r+\lambda \Phi \left(\eta \right)\right)h\left(\eta \right) \tilde{P}\left(\mathrm{d}\eta \right)\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & -{\int }_{\mathcal{X}}\frac{e^{\Phi \left(\eta \right)}{\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)}h\left({\eta }^{\prime }\right) \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)}{{\left({\int }_{\mathcal{X}}{e}^{\Phi \left({\eta }^{\prime }\right)} \tilde{P}\left(\mathrm{d}{\eta }^{\prime }\right)\right)}^2}\left(d{\left(\eta \right)}^r+\lambda \Phi \left(\eta \right)\right) \tilde{P}\left(\mathrm{d}\eta \right).\hfill \end{array}$$

(38)

Per (37) and (38), the derivative vanishes for every function h if $d{\left(\eta \right)}^r+\lambda \Phi \left(\eta \right)=0$. As $\xi$ is arbitrary, the general minimum is attained for $\Phi (\xi ,\eta )=-d{(\xi ,\eta )}^r/\lambda$. With this, the first expression in (31) vanishes, and we can conclude that

$$\begin{array}{cc}\hfill \underset{\pi }{inf}{\mathbb{E}}_{\pi }{d}^r+\lambda D(\pi \parallel P\times \tilde{P})& =-\lambda {\int }_{\mathcal{X}}log{\int }_{\mathcal{X}}{e}^{-d{(\xi ,\eta )}^r/\lambda } \tilde{P}\left(\mathrm{d}\eta \right)P\left(\mathrm{d}\xi \right)\hfill \\ \hfill & ={\mathbb{E}}_P\left({min}_{\tilde{P};\lambda }d{(\xi ,\tilde{\xi })}^r\right).\hfill \end{array}$$

Finally, notice that the variable $Z(\xi ,\eta )=e^{\Phi (\xi ,\eta )}$ is completely arbitrary for the problem in (26) involving the Wasserstein distance and the Kullback–Leibler divergence. As outlined above, for every measure $\pi$ with finite divergence $D(\pi \parallel P\times \tilde{P})$, there is a density Z, as considered above. From this, the assertion in Proposition 1 follows.    □

Remark 5.

The preceding proposition considers probability measures π with marginal ${\pi }_1=P$. Its first marginal distribution (trivially) is absolutely continuous with respect to P, ${\pi }_1\ll P$, as ${\pi }_1=P$.

The second marginal ${\pi }_2$, however, is not specified. In order for π to be feasible in (26), its Kullback–Leibler divergence with respect to $P\times \tilde{P}$ must be finite. Hence, there is a (non-negative) Radon–Nikodým density Z such that

$${\pi }_2\left(B\right)=\pi (\mathcal{X}\times B)=\underset{\mathcal{X}\times B}{\int \int }Z(\xi ,\eta )P\left(\mathrm{d}\xi \right)\tilde{P}\left(\mathrm{d}\eta \right).$$

It follows from Fubini’s theorem that

$${\pi }_2\left(B\right)={\int }_B{\int }_{\mathcal{X}}Z(\xi ,\eta ) P\left(\mathrm{d}\xi \right)\tilde{P}\left(\mathrm{d}\eta \right)={\int }_{B}Z\left(\eta \right) \tilde{P}\left(\mathrm{d}\eta \right),$$

where $Z\left(\eta \right):={\int }_{\mathcal{X}}Z(\xi ,\eta ) P\left(\mathrm{d}\xi \right)$. Thus, the second marginal is absolutely continuous with respect to $\tilde{P}$, ${\pi }_2\ll \tilde{P}$.

Proposition 1 characterizes the objective of the quantization problem. In addition, its proof implicitly reveals the marginal of the best approximation. The following lemma explicitly spells out the density of the marginal of the optimal measure with respect to $\tilde{P}$.

Lemma 2

(Characterization of the best approximating measure). The best approximating marginal probability measure minimizing (26) has a density

$$Z\left(\tilde{\xi }\right)={\mathbb{E}}_P{\sigma }_{\lambda }\left(d{(\xi ,\tilde{\xi })}^r\right)={\int }_{\mathcal{X}}{\sigma }_{\lambda }\left(d{(\xi ,\tilde{\xi })}^r\right) P\left(\mathrm{d}\xi \right),$$

where ${\sigma }_{\lambda }(·)$ is the softmin function (cf. Definition 4).

Proof. 

Recall from the proof of Proposition 1 that we have the density

$$\tilde{Z}(\xi ,\tilde{\xi })=\frac{e^{-d{(\xi ,\tilde{\xi })}^r/\lambda }}{{\mathbb{E}}_{\tilde{P}}{e}^{-d{(\xi ,\tilde{\xi })}^r/\lambda }}$$

of the optimal measure $\pi$ relative to $P\times \tilde{P}$. From this, we can derive

$${\pi }_2\left(B\right)=\pi (\mathcal{X}\times B)={\int }_B{\int }_{\mathcal{X}}\frac{e^{-d{(\xi ,\tilde{\xi })}^r/\lambda }}{{\mathbb{E}}_{\tilde{P}}{e}^{-d{(\xi ,\tilde{\xi })}^r/\lambda }} P\left(\mathrm{d}\xi \right)\tilde{P}\left(\mathrm{d}\tilde{\xi }\right)$$

such that

$$Z\left(\tilde{\xi }\right)={\int }_{\mathcal{X}}\frac{e^{-d{(\xi ,\tilde{\xi })}^r/\lambda }}{{\mathbb{E}}_{\tilde{P}}{e}^{-d{(\xi ,\tilde{\xi })}^r/\lambda }}P\left(\mathrm{d}\xi \right)={\mathbb{E}}_P{\sigma }_{\lambda }\left(d{(\xi ,\tilde{\xi })}^r\right)$$

is the density with respect to $\tilde{P}$, that is, $\mathrm{d}{\pi }_2=Z \mathrm{d}\tilde{P}$ (i.e., ${\pi }_2\left(\mathrm{d}\tilde{\xi }\right)=Z\left(\tilde{\xi }\right) \tilde{P}\left(\mathrm{d}\tilde{\xi }\right)$).    □

3.2. Approximation with Flexible Marginal Measure

The following proposition reveals that the best approximation of a bivariate measure in terms of a product of independent measures is provided by the product of its marginals. With this, it follows that the objectives in (25) and (26) coincide for $\tilde{P}={\pi }_2$.

Proposition 2.

Let P be a measure and let π be a bivariate measure with marginal ${\pi }_1=P$ and ${\pi }_2$. Then, it holds that

$$D(\pi \parallel P\times {\pi }_2)\le D(\pi \parallel P\times \tilde{P}),$$

(39)

where $\tilde{P}$ is an arbitrary measure.

Proof. 

Define the Radon–Nikodým density $Z\left(\eta \right):=\frac{{\pi }_2\left(\mathrm{d}\eta \right)}{\tilde{P}\left(\mathrm{d}\eta \right)}$ and observe that the extension $Z(\xi ,\eta ):=Z\left(\eta \right)$ to $\mathcal{X}\times \mathcal{X}$ is the density $Z=\frac{\mathrm{d}P\times {\pi }_2}{\mathrm{d}P\times \tilde{P}}$. It follows with (4) that

$$\begin{array}{cc}\hfill 0\le D({\pi }_2\parallel \tilde{P})& ={\mathbb{E}}_{{\pi }_2}log\frac{\mathrm{d}{\pi }_2}{\mathrm{d}\tilde{P}}\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_{\pi }log\frac{\mathrm{d} P\times {\pi }_2}{\mathrm{d} P\times \tilde{P}}\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_{\pi }\left(log\frac{\mathrm{d} \pi }{\mathrm{d} P\times \tilde{P}}-log\frac{\mathrm{d} \pi }{\mathrm{d} P\times {\pi }_2}\right)\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill & =D(\pi \parallel P\times \tilde{P})-D(\pi \parallel P\times {\pi }_2),\hfill \end{array}$$

(43)

which is the assertion. In case the measures are not absolutely continuous, the assertion in (40) is trivial.    □

Suppose now that $\pi$ is a solution of the master problem (26) with some $\tilde{P}$. It follows from the preceding proposition that the objective (26) improves when replacing the initial $\tilde{P}$ with the marginal of the optimal solution, that is, $\tilde{P}={\pi }_2$.

3.3. The Relation of Soft Quantization and Entropy

The soft quantization problem (26) involves the Kullback–Leibler divergence and not the entropy. The major advantage of the formulation presented above is that it works for discrete, continuous, or mixed measures, while entropy usually needs to be defined separately for discrete and continuous measures.

For a discrete measure with $P\left(x\right):=P\left(\right\{x\left\}\right)$ and $\tilde{P}\left(y\right):=\tilde{P}\left(\left\{y\right\}\right)$, the Kullback–Leibler divergence (4) is

$$\begin{array}{cc}\hfill D(\tilde{P}\parallel P)& =H(\tilde{P},P)-H\left(\tilde{P}\right)\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill & =\sum _{x\in \mathcal{X}}\tilde{P}\left(x\right)log\frac{\tilde{P}\left(x\right)}{P\left(x\right)},\hfill \end{array}$$

(45)

where

$$H(\tilde{P},P):=-\sum _{x\in \mathcal{X}}\tilde{P}\left(x\right)·logP\left(x\right)$$

is the cross-entropy of the measures $\tilde{P}$ and P, while 

$$H\left(\tilde{P}\right):=H(\tilde{P},\tilde{P})=-\sum _{x\in \mathcal{X}}\tilde{P}\left(x\right)log\tilde{P}\left(x\right)$$

(46)

is the entropy of $\tilde{P}$.

For a measure $\pi$ with marginals P and $\tilde{P}$, the cross-entropy is

$$\begin{array}{cc}\hfill H(\pi ,P\times \tilde{P})& =-\sum _{x,y}\pi (x,y)log\left(P\left(x\right)·\tilde{P}\left(y\right)\right)\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill & =-\sum _{x,y}\pi (x,y)logP\left(x\right)-\sum _{x,y}\pi (x,y)log\tilde{P}\left(y\right)\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill & =-\sum _{x}P\left(x\right)logP\left(x\right)-\sum _y\tilde{P}\left(y\right)log\tilde{P}\left(y\right),\hfill \end{array}$$

(49)

where we have used the marginals from (2). Note that (49) does not depend on $\pi$; hence, $H(\pi ,P\times \tilde{P})$ does not depend on $\pi$.

With (44), the quantization problem (26) can be rewritten equivalently as

$$\underset{\pi :{\pi }_2\in P}{min}\underset{\mathcal{X}\times \mathcal{X}}{\int \int }{d}^r \mathrm{d}\pi -\lambda ·H\left(\pi \right)$$

(50)

by involving the entropy only. For this reason, we call the master problem in (26) the entropy-regularized problem.

4. Soft Tessellation

The quantization problem (25) consists of finding a good (in the best case, the optimal) approximation of a general probability measure P on $\mathcal{X}$ using a simple and discrete measure ${\tilde{P}}_m={\sum }_{j=1}^m{\tilde{p}}_j {\delta }_{y_j}$. Thus, the problem consists of finding good weights ${\tilde{p}}_1,\dots ,{\tilde{p}}_m$ as well as good locations $y_1,\dots ,y_m$. Quantization employs the Wasserstein distance to measure the quality of the approximation; instead, soft quantization involves the regularized Wasserstein distance, as in (26):

$$\underset{{\tilde{P}}_m\in {\mathcal{P}}_m\left(\mathcal{X}\right)}{inf} \underset{\pi :\begin{array}{c}{\pi }_1=P,\\ {\pi }_2={\tilde{P}}_m\end{array}}{inf}{\mathbb{E}}_{\pi }{d}^r+\lambda ·D(\pi \parallel P\times {\tilde{P}}_m),$$

where the measures on $\mathcal{X}$ supported by not more than m points (cf. (23)) are as follows:

$${\mathcal{P}}_m\left(\mathcal{X}\right)=\left\{{\tilde{P}}_m\in \mathcal{P}\left(\mathcal{X}\right):{\tilde{P}}_m=\sum _{j=1}^m{\tilde{p}}_j {\delta }_{y_j}\right\}.$$

We separate the problems of finding the best weights and locations. The following Section 4.1 addresses the problem of finding the optimal weights $\tilde{p}$; the subsequent Section 4.2 then addresses the problem of finding the optimal locations $y_1,\dots ,y_m$. As well, we elaborate the numerical advantages of soft quantization below.

4.1. Optimal Weights

Proposition 1 above is formulated for the general probability measures P and $\tilde{P}$. The desired measure in quantization is a simple and discrete measure. To this end, recall that, per Remark 5, measures which are feasible for (26) have marginals ${\pi }_2$ with ${\pi }_2\ll \tilde{P}$. It follows that the support of the marginal is smaller than the support of $\tilde{P}$, that is,

$$\mathrm{supp}{\pi }_2\subset \mathrm{supp}\tilde{P}.$$

For a simple measure $\tilde{P}={\sum }_{j=1}^m{\tilde{p}}_j {\delta }_{y_j}$ with ${\tilde{p}}_j>0$, it follows in particular that $\mathrm{supp}{\pi }_2\subset \{y_1,\dots ,y_m\}$. In this subsection, we consider the measure $\tilde{P}$ and the support $\{y_1,\dots ,y_m\}$ to be fixed.

To unfold the result of Proposition 1 for discrete measures, recall the smooth minimum and the softmin function for the discrete (empirical or uniform) measure $\tilde{P}={\sum }_{j=1}^m{\tilde{p}}_j {\delta }_{y_j}$. For this measure, the smooth minimum (6) explicitly is

$${min}_{\lambda ;\tilde{P}}(y_1,\dots ,y_m)=-\lambda log\left({\tilde{p}}_1 e^{-y_1/\lambda }+\dots +{\tilde{p}}_m e^{-y_m/\lambda }\right).$$

This function is occasionally referred to as the LogSumExp function. The softmin function (or Gibbs density (12)) is

$${\sigma }_{\lambda }(y_1,\dots ,y_m)={\left(\frac{e^{-y_j/\lambda }}{{\tilde{p}}_1 e^{-y_1/\lambda }+\dots +{\tilde{p}}_m e^{-y_m/\lambda )}}\right)}_{j=1}^m.$$

It follows from Lemma 2 that the best approximating measure is $Q={\sum }_{j=1}^m{q}_j {\tilde{p}}_j {\delta }_{y_j}$, where the vector q of the optimal weights relative to $\tilde{P}$ is provided explicitly by

$$q={\int }_{\mathcal{X}}{\sigma }_{\lambda }\left(d{(\xi ,y_1)}^r,\dots ,d{(\xi ,y_m)}^r\right) P\left(\mathrm{d}\xi \right)={\mathbb{E}}_P{\sigma }_{\lambda }\left(d{(\xi ,y_1)}^r,\dots ,d{(\xi ,y_m)}^r\right),$$

(51)

which involves computing expectations.

• Soft Tessellation

For $\lambda =0$, the softmin function ${\sigma }_{\lambda }$ is

$${\tilde{p}}_j·{\sigma }_{\lambda =0}{\left(d{(\xi ,y_1)}^r,\dots ,d{(\xi ,y_m)}^r\right)}_j=\left\{\begin{array}{cc}1\hfill & \mathrm{if}d{(\xi ,y_j)}^r=min\left(d{(\xi ,y_1)}^r,\dots ,d{(\xi ,y_m)}^r\right),\hfill \\ 0\hfill & \mathrm{else}.\hfill \end{array}\right.$$

That is, the mapping $j\mapsto {\tilde{p}}_j·{\sigma }_{\lambda }{(\dots )}_j$ can serve for classification, i.e., tessellation; the point $\xi$ is associated with $y_j$ if ${\sigma }_{\lambda }{(\dots )}_j\ne 0$, and the corresponding region is known as a Voronoi diagram.

For $\lambda >0$, the softmin ${\tilde{p}}_j·{\sigma }_{\lambda }{(\dots )}_j$ is not a strict indicator, and can instead be interpreted as probability; that is,

$${\tilde{p}}_j·{\sigma }_{\lambda }{\left(d{(\xi ,y_1)}^r,\dots ,d{(\xi ,y_m)}^r\right)}_j$$

is the probability of allocating $\xi \in \mathcal{X}$ to the quantizer $y_j$.

Remark 6

(K-means and quantization). K-means clustering is a widely used unsupervised machine learning algorithm that groups datapoints into clusters based on their similarity. Voronoi tessellation, on the other hand, is a geometrical concept used to partition a space into regions, each of which are associated with a specific point or seed. Notably, this is an unavoidable concept in the investigation of optimal quantizers. In the K-means context, Voronoi tessellation helps to define cluster boundaries. Each cluster’s boundary is constructed as the region within which the datapoints are closer to its cluster center than to any other (cf. Graf and Luschgy [4] Chapter I).

4.2. Optimal Locations

As a result of Proposition 1, the objective in (27) is an expectation. To identify the optimal support points $y_1,\dots ,y_m$, it is essential to first minimize

$$\underset{\tilde{P}={\sum }_{j=1}^m{\tilde{p}}_j{\delta }_{y_j}}{min}{\mathbb{E}}_{\xi \sim P}\left({min}_{\lambda ;y\sim \tilde{P}}d{(\xi ,y)}^r\right).$$

(52)

This is a stochastic, nonlinear, and non-convex optimization problem:

$$f(y_1,\dots ,y_m):=\mathbb{E}f(y_1,\dots ,y_m;\xi )=\mathbb{E}\left(\underset{j=1,\dots ,m}{{min}_{\lambda ;\tilde{P}}}d{(\xi ,y_i)}^r\right),$$

(53)

where the function $f(y_1,\dots ,y_m;\xi ):={min}_{\lambda ;\tilde{P}}\{d{(\xi ,y_i)}^r:j=1,\dots ,m\}$ is nonlinear and non-convex. The optimal quantization problem constitutes an unconstrained, stochastic, non-convex, and nonlinear optimization problem. According to the chain rule and gradients in Section 2.3, the gradient of the objective is constructed from the components

$$\frac{\partial }{\partial y_j}f(y_1,\dots ,y_m)=\frac{{\tilde{p}}_j·exp\left(-d{(\xi ,y_j)}^r/\lambda \right)}{{\sum }_{j^{\prime }=1}^m{\tilde{p}}_{j^{\prime }}·exp\left(-d{(\xi ,y_{j^{\prime }})}^r/\lambda \right)}·{\left.{\nabla }_y d{(\xi ,y)}^r\right|}_{y=y_j},$$

(54)

that is,

$$\nabla f=\tilde{p}·{\sigma }_{\lambda }\left(d{(\xi ,y_1)}^r,\dots ,d{(\xi ,y_m)}^r\right)·r d{(\xi ,y)}^{r-1}·{\nabla }_y d(\xi ,y),$$

(55)

where ‘·’ denotes the Hadamard (element-wise) product and $\tilde{p}$, $d{(\xi ,y)}^{r-1}$ are the vectors with entries ${\tilde{p}}_j$, $d{(\xi ,y_j)}^{r-1}$, $j=1,\dots ,m$. In other words, the gradient of the LogSumExp function (53) is the softmin function, which Section 2.3 explicitly illustrates.

Algorithm 1 is a stochastic gradient algorithm used to minimize (51), which collects the elements of the optimal weights and the optimal locations provided here and in the preceding section.

Algorithm 1: Stochastic gradient algorithm to find the optimal quantizers and optimal masses

Example 1.

To provide an example of the gradient of the distance function in (54) ((55), resp.), the derivative of the weighted norm

$$d(\xi ,y)={\parallel y-\xi \parallel }_p:={\left(\sum _{\ell =1}^d{w}_{\ell }·{|y_{\ell }-{\xi }_{\ell }|}^p\right)}^{1/p}$$

is

$$\frac{\partial }{\partial y_j}{\parallel y-\xi \parallel }_p^r=r w_j{ \parallel \xi -y\parallel }_p^{\frac{r-p}{p}}·{|y_j-{\xi }_j|}^{p-1}·\mathrm{sign}(y_j-{\xi }_j).$$

4.3. Quantization with Large Regularization Parameters

The entropy in (46) is minimal for the Dirac measure $P={\delta }_x$ (where x is any point in $\mathcal{X}$); in this case, $H\left({\delta }_x\right)=1·log1=0$, while $H\left(\tilde{P}\right)>0$ for any other measure. For larger values of $\lambda$, the objective in (50), and as such the objective of the master problem (23), will supposedly prefer a measure with fewer points. This is indeed the case, as stated by Theorem 1 above. We provide its proof below after formally defining the center of the measure.

Definition 5

(Center of the measure). Let P be a probability measure on $\mathcal{X}$ and let d be a distance on $\mathcal{X}$. The point $a\in \mathcal{X}$ is a center of the measure P with respect to the distance d if

$$a\in \underset{x\in \mathcal{X}}{arg\; min}\mathbb{E}d{(x,\xi )}^r,$$

provided that $\mathbb{E}d{(x_0,\xi )}^r<\infty$ for some (i.e., any) $x_0\in \mathcal{X}$ and $r\ge 1$.

In what follows, we demonstrate that the regularized quantization problem (50) links the optimal quantization problem and the center of the measure.

Proof of Theorem 1.

According to Proposition 1, Problems (50) and (26) are equivalent. Now, assume that $y_i=y_j$ for all i, $j\le m$; then, $d(y_i,\xi )=d(y_j,\xi )$ for $\xi \in \Xi$, and it follows that

$${min}_{\lambda }\left(d{(y_1,\xi )}^r,\dots ,d{(y_m,\xi )}^r\right)=d{(y_i,\xi )}^r, i=1,\dots ,m.$$

Thus, the minimum of the optimization problem is attained at $y_i=a$ for each $i=1,\dots ,m$, where a is the center of the measure P with respect to the distance d. It follows that $y_1=\dots =y_m=a$ is a local minimum and a stationary point satisfying the first order conditions

$$\nabla f(y_1,\dots ,y_m)=0$$

for the function f provided in (53). Note as well that

$${\sigma }_{\lambda }{\left(d{(\xi ,y_1)}^r,\dots ,d{(\xi ,y_n)}^r\right)}_i=\frac{exp\left(-d{(\xi ,y_i)}^r/\lambda \right)}{{\sum }_{j=1}^n{\tilde{p}}_{j}exp\left(-d{(\xi ,y_j)}^r/\lambda \right)}=1,$$

and as such the softmin function does not depend on $\lambda$ at the stationary point $y_1=\dots =y_m=a$.

Recall from (54) that

$$\nabla \mathbb{E}\left(\underset{j=1,\dots ,n}{{min}_{\lambda ;\tilde{P}}}d{(\xi ,y_j)}^r\right)=\mathbb{E}{\sigma }_{\lambda }\left(d{(y_1,\xi )}^r,\dots ,d{(y_n,\xi )}^r\right)·\nabla d{(\xi ,y_i)}^r.$$

According to the product rule, the Hessian matrix is

$${\nabla }^2\mathbb{E}\left(\underset{j=1,\dots ,n}{{min}_{\lambda ;\tilde{P}}}d{(\xi ,y_j)}^r\right)=\mathbb{E}\left(\begin{array}{c}\begin{array}{c}\nabla {\sigma }_{\lambda }\left(d{(y_1,\xi )}^r,\dots ,d{(y_n,\xi )}^r\right)·{\left(\nabla d{(\xi ,y_i)}^r\right)}^2\hfill \\ \hspace{1em}+{\sigma }_{\lambda }\left(d{(y_1,\xi )}^r,\dots ,d{(y_n,\xi )}^r\right)·{\nabla }^{2}d{(\xi ,y_i)}^r\hfill \end{array}\end{array}\right).$$

(56)

Note that the second expression is positive definite, as the Hessian ${\nabla }^{2}d{(\xi ,y_i)}^r$ of the convex function is positive definite and $\nabla {{min}_{\lambda ;\tilde{P}}}_{j=1,\dots ,n}(x_1,\dots ,x_n)={\sigma }_{\lambda }(x_1,\dots ,x_n)\ge 0$. Further, the Hessian of the smooth minimum (see Appendix A) is

$$\nabla {\sigma }_{\lambda }={\nabla }^2\underset{j=1,\dots ,n}{{min}_{\lambda }}=-\frac{1}{\lambda } \Sigma ,$$

where the matrix $\Sigma$ is

$$\Sigma :=diag\left({\sigma }_1,\dots ,{\sigma }_n\right)-\sigma {\sigma }^{\top }.$$

This matrix $\Sigma$ is positive definite (as ${\sum }_{i=1}^n{\sigma }_i=1$) and $0\le \Sigma \le 1$ in Loewner order; indeed, $\Sigma$ is the covariance matrix of the multinomial distribution. It follows that the first term in (56) is $\mathcal{O}\left(1\right)$, while the second is $\mathcal{O}\left(\frac{1}{\lambda }\right)$, such that (56) is positive definite for sufficiently small $\lambda$. Thus, the extremal point $y_i=a$ is a minimum for all $\lambda$. In particular, there exists ${\lambda }_0>0$ such that (56) is positive definite for every $\lambda >{\lambda }_0$, hence, the result. □

5. Numerical Illustration

This section presents numerical findings for the approaches and methods discussed earlier. The Julia implementations for these methods are available online (cf. https://github.com/rajmadan96/SoftQuantization.git, accessed on 8 September 2023).

In the following experiments, we approximate the measure P with a finite discrete measure $\tilde{P}$ using the stochastic gradient algorithm presented in Algorithm 1.

5.1. One Dimension

First, we perform the analysis in one dimension. In this experiment, our problem of interest is to find entropy-regularized optimal quantizers for

$$P\sim \mathcal{N}(0,1)\mathrm{and}P\sim Exp\left(1\right)$$

(i.e., the normal and exponential distributions with standard parameters). To enhance the peculiarity, we consider only $m=8$ quantizers.

Figure 1 illustrates the results of soft quantization of the standard normal distribution and exponential distribution. It is apparent that when $\lambda$ is increased beyond a certain threshold (cf. Theorem 1), the quantizers converge towards the center of the measure (i.e., the mean), while for smaller values of $\lambda$ the quantizers are able to identify the actual optimal locations with greater accuracy. Furthermore, we emphasize that our proposed method is capable of identifying the mean location regardless of the shape of the distribution, which this experiment empirically substantiates.

For better understanding of the dissemination of the weights (probabilities) and their respective positions, the following examination involves the calculation of the cumulative distribution function. Additionally, we consider

$$P\sim \Gamma (2,2) (\mathrm{Gamma}\mathrm{distribution})$$

as a problem of interest, which is a notably distinct scenario in terms of shape compared to the measures examined previously.

Figure 2 provides the results. It is evident that the number of quantizers m decreases as $\lambda$ increases. When $\lambda$ reaches a specific threshold, such as $\lambda =20$ in our case, all quantizers converge towards the center of the measures, represented by the mean (i.e., 4).

5.2. Two Dimensions

Next, we demonstrate the behavior of entropy-regularized optimal quantization for a range of $\lambda$ in two dimensions. In the following experiment, we consider

$$P\sim U\left((0,1)\times (0,1)\right) (\mathrm{uniform}\mathrm{distribution}\mathrm{on}\mathrm{the}\mathrm{square})$$

as a problem of interest. Initially, we perform the experiment with $m=4$ quantizers.

Figure 3 illustrates the findings. Figure 3a reveals a quantization pattern similar to that observed in the one-dimensional experiment. However, in Figure 3b we gain more detailed insight into the behavior of the quantizers at $\lambda =1$, where they align diagonally before eventually colliding. Furthermore, the size of the point indicates the respective probability of the quantization point, which is notably uniformly distributed for a varying regularization parameter $\lambda$.

Again, we consider a uniform distribution as a problem of interest in the subsequent experiment, this time employing $m=16$ quantizers for enhanced comprehension. Figure 4 encapsulates the essence of the experiment, offering an extensive visual representation. In contrast to the previous experiment, it can be observed that for regularization values of $\lambda =0.037$ and $\lambda =0.1$ they assemble at the nearest strong points (in terms of high probability) rather than converging towards the center of the measure (see Figure 4b,c). Subsequently, for larger $\lambda$, they move from these strong points towards the center, where they are in a diagonal alignment before colliding (see Figure 4d). More concisely, when $\lambda =0$ we achieve the genuine quantization solution (see Figure 4a). As $\lambda$ increases, the quantizers with lower probabilities converge towards those with nearest higher probabilities. Subsequently, all quantizers converge towards the center of the measure, represented by the mean of respective measure.

Thus far, we have conducted two-dimensional experiments employing various quantizers ($m=4$ and $m=16$) with the uniform distribution. These experiments can be categorized under the k-means approach (see Remark 6). Next, we delve into the complexity of a multivariate normal distribution, with the aim of enhancing comprehension. More precisely, our problem of interest is to find a soft quantization for

$$P\sim \mathcal{N}(\mu ,\Sigma ),$$

where

$$\mu =\left(\begin{array}{c}0\\ 0\end{array}\right),\Sigma =\left(\begin{array}{cc}3& 1\\ 1& 3\end{array}\right).$$

In this endeavor, we employ more quantizers, specifically, $m=100$. Figure 5 captures the core essence of the experiment, delivering a comprehensive and visually illustrative representation. From the experiment, it is evident that the initial diagonal alignment precedes convergence toward the center of the measure as $\lambda$ increases. Additionally, a noticeable shift can be observed on the part of the points with lower probabilities towards those with higher probabilities. This experiment highlights that the threshold of $\lambda$ for achieving convergence or diagonal alignment in the center of the measure depends on the number of quantizers employed.

6. Summary

In this study, we have enhanced the stability and simplicity of the standard quantization problem by introducing a novel method of quantization using entropy. Propositions 1 and 2 thoroughly elucidate the intricacies of the master problem (25). Our substantiation of the convergence of quantizers to the center of the measure explains the transition from a complex hard optimization problem to a simplified configuration (see Theorem 1). More concisely, this transition underscores the fundamental shift towards a more tractable and straightforward computational framework, marking a significant advancement in terms of the overall approach. Moreover, in Section 5, we provide numerical illustrations of our method that confirm its robustness, stability, and properties, as discussed in our theoretical results. These numerical demonstrations serve as empirical evidence reinforcing the efficacy of our proposed approach.