Hardness and Approximability of Dimension Reduction on the Probability Simplex

Abstract

Dimension reduction is a technique used to transform data from a high-dimensional space into a lower-dimensional space, aiming to retain as much of the original information as possible. This approach is crucial in many disciplines like engineering, biology, astronomy, and economics. In this paper, we consider the following dimensionality reduction instance: Given an n-dimensional probability distribution p and an integer $m<n$, we aim to find the m-dimensional probability distribution q that is the closest to p, using the Kullback–Leibler divergence as the measure of closeness. We prove that the problem is strongly NP-hard, and we present an approximation algorithm for it.

1. Introduction

Dimension reduction [1,2] is a methodology for mapping data from a high-dimensional space to a lower-dimensional space, while approximately preserving the original information content. This process is essential in fields such as engineering, biology, astronomy, and economics, where large datasets with high-dimensional points are common.

It is often the case that the computational complexity of the algorithms employed to extract relevant information from these datasets depends on the dimension of the space where the points lie. Therefore, it is important to find a representation of the data in a lower-dimensional space that still (approximately) preserves the information content of the original data, as per given criteria.

A special case of the general issue illustrated before arises when the elements of the dataset are n-dimensional probability distributions, and the problem is to approximate them by lower-dimensional ones. This question has been extensively studied in different contexts. In [3,4], the authors address the problem of dimensionality reduction on sets of probability distributions with the aim of preserving specific properties, such as pairwise distances. In [5], Gokhale considers the problem of finding the distribution that minimizes, subject to a set of linear constraints on the probabilities, the “discrimination information” with respect to a given probability distribution. Similarly, in [6], Globerson et al. address the dimensionality reduction problem by introducing a nonlinear method aimed at minimizing the loss of mutual information from the original data. In [7], Lewis explores dimensionality reduction for reducing storage requirements and proposes an approximation method based on the maximum entropy criterion. Likewise, in [8], Adler et al. apply dimensionality reduction to storage applications, focusing on the efficient representation of large-alphabet probability distributions. More closely related to the dimensionality reduction that we deal with in this paper are the works [9,10,11,12]. In [10,11], the authors address task scheduling problems where the objective is to allocate tasks of a project in a way that maximizes the likelihood of completing the project by the deadline. They formalize the problem in terms of random variables approximation by using the Kolmogorov distance as a measure of distance and present an optimal algorithm for the problem. In contrast, in [12], Vidyasagar defines a metric distance between probability distributions on two distinct finite sets of possibly different cardinalities based on the Minimum Entropy Coupling (MEC) problem. Informally, in the MEC, given two probability distributions p and q, one seeks to find a joint distribution $\varphi$ that has p and q as marginal distributions and also has minimum entropy. Unfortunately, computing the MEC is NP-hard, as shown in [13]. However, numerous works in the literature present efficient algorithms for computing couplings with entropy within a constant number of bits from the optimal value [14,15,16,17,18]. We note that computing the coupling of a pair of distributions can be seen as essentially the inverse of dimension reduction. Specifically, given two distributions p and q, one constructs a third, larger distribution $\varphi$, such that p and q are derived from $\varphi$ or, more formally, aggregations of $\varphi$. In contrast, the dimension reduction problem addressed in this paper involves starting with a distribution p and creating another, smaller distribution that is derived from p or, more formally, is an aggregation of p.

Moreover, in [12], the author demonstrates that, according to the defined metric, any optimal reduced-order approximation must be an aggregation of the original distribution. Consequently, the author provides an approximation algorithm based on the total variation distance, using an approach similar to the one we will employ in Section 4. Similarly, in [9], Cicalese et al. examine dimensionality reduction using the same distance metric introduced in [12]. They propose a general criterion for approximating p with a shorter vector q, based on concepts from Majorization theory, and provide an approximation approach to solve the problem.

We also mention that analogous problems arise in scenario reduction [19], where the problem is to (best) approximate a given discrete distribution with another distribution with fewer atoms in compressing probability distributions [20] and elsewhere [21,22,23]. Moreover, we recommend the following survey for further application examples [24].

In this paper, we study the following instantiation of the general problem described before: Given an n-dimensional probability distribution $p=(p_1,\dots ,p_n)$, and $m<n$, find the m-dimensional probability distribution $q=(q_1,\dots ,q_m)$ that is the closest to p, where the measure of closeness is the well-known relative entropy [25] (also known as Kullback–Leibler divergence). In Section 2, we formally state the problem. In Section 3, we prove that the problem is strongly NP-hard, and in Section 4, we provide an approximation algorithm returning a solution whose distance from p is at most 1 plus the minimum possible distance.

2. Statement of the Problem and Mathematical Preliminaries

Let

$${\mathcal{P}}_n=\{p=(p_1,\cdots ,p_n)\mid p_1\ge \cdots \ge p_n>0,\sum _{i=1}^n{p}_i=1\}$$

(1)

be the $(n-1)$-dimensional probability simplex. Given two probability distributions $p\in {\mathcal{P}}_n$ and $q\in {\mathcal{P}}_m$, with $m<n$, we say that q is an aggregation of p if each component of q can be expressed as the sum of distinct components of p. More formally, q is an aggregation of p if there exists a partition $\mathrm{\Pi }=({\mathrm{\Pi }}_1,\dots ,{\mathrm{\Pi }}_m)$ of $\{1,\dots ,n\}$ such that $q_i={\sum }_{j\in {\mathrm{\Pi }}_i}{p}_j$, for each $i=1,\dots ,m$. Notice that the aggregation operation corresponds to the following operation on random variables: Given a random variable X that takes value in a finite set $\mathcal{X}=\{x_1,\dots ,x_n\}$, such that $Pr\{X=x_i\}=p_i$ for $i=1,\dots ,n$, any function $f:\mathcal{X}\mapsto \mathcal{Y}$, with $\mathcal{Y}=\{y_1,\cdots ,y_m\}$ and $m<n$, induces a random variable $f\left(X\right)$ whose probability distribution $q=(q_1,\dots ,q_m)$ is an aggregation of p. Dimension reduction in random variables through the application of deterministic functions is a common technique in the area (e.g., [10,12,26]). Additionally, the problem arises also in the area of “hard clustering” [27] where one seeks a deterministic mapping f from data, generated by an r.v. X taking values in a set $\mathcal{X}$, to “labels” in some set $\mathcal{Y}$, where typically $\left|\mathcal{Y}\right|\ll \left|\mathcal{X}\right|$.

For any probability distribution $p\in {\mathcal{P}}_n$ and an integer $m<n$, let us denote by ${\mathcal{A}}_m\left(p\right)$ the set of all $q\in {\mathcal{P}}_m$ that are aggregations of p. Our goal is to solve the following optimization problem:

Problem 1.

Given $p\in {\mathcal{P}}_n$ and $m<n$, find $q^{*}\in {\mathcal{A}}_m\left(p\right)$ such that

$$\underset{q\in {\mathcal{A}}_m\left(p\right)}{min}D(q\parallel p)=D(q^{*}\parallel p),$$

(2)

where $D(q\parallel p)$ is the relative entropy [25], given by

$$D(q\parallel p)=\sum _{i=1}^m{q}_{i}log\frac{q_i}{p_i},$$

and the logarithm is of base 2.

An additional motivation to study Problem 1 comes from the fundamental paper [28], in which the principle of minimum relative entropy (called therein minimum cross entropy principle) is derived in an axiomatic manner. The principle states that, of the distributions q that satisfy given constraints (in our case, that $q\in {\mathcal{A}}_m\left(p\right)$), one should choose the one with the least relative entropy “distance” from the prior p.

Before establishing the computational complexity of the Problem 1, we present a simple lower bound on the optimal value.

Lemma 1.

For each $p\in {\mathcal{P}}_n$ and $q\in {\mathcal{P}}_m$, $m<n$, it holds that

$$D(q\parallel p)\ge D(lb\left(p\right)\parallel p)=-log\left(\sum _{i=1}^m{p}_i\right).$$

(3)

where

$$lb\left(p\right)=\left(\frac{p_1}{{\sum }_{i=1}^m{p}_i},\cdots ,\frac{p_m}{{\sum }_{i=1}^m{p}_i}\right)\in {\mathcal{P}}_m.$$

(4)

Proof. 

Given an arbitrary $p\in {\mathcal{P}}_n$, one can see that

$$D(lb\left(p\right)\parallel p)=-log\left(\sum _{i=1}^m{p}_i\right).$$

Moreover, for any $p\in {\mathcal{P}}_n$ and $q\in {\mathcal{P}}_m$, the Jensen inequality applied to the log function gives the following:

$$-D\left(q\right|\left|p\right)=\sum _{i=1}^m{q}_{i}log\frac{p_i}{q_i}\le log\left(\sum _{i=1}^m{p}_i\right).$$

   □

3. Hardness

In this section, we prove that the optimization problem (2) described in Section 1 is strongly NP-hard. We accomplish this by reducing the problem from the 3-Partition problem, a well-known strongly NP-hard problem [29], described as follows.

3-Partition: Given a multiset $S=\{a_1,\cdots ,a_n\}$ of $n=3m$ positive integers for which ${\sum }_{i=1}^n{a}_i=mT$, for some T, the problem is to decide whether S can be partitioned into m triplets such that the sum of each triple is exactly T. More formally, the problem is to decide whether there exist $S_1,\cdots ,S_m\subseteq S$ such that the following conditions hold:

$$\begin{array}{cc}\hfill \sum _{a\in S_j}a& =T,\forall j\in \{1,\cdots ,m\},\hfill \\ \hfill S_i\cap S_j& =\varnothing ,\forall i\ne j,\hfill \\ \hfill \bigcup _{i=1}^m{S}_i& =S,\hfill \\ \hfill |S_i|& =3,\forall i\in \{1,\cdots ,m\}.\hfill \end{array}$$

Theorem 1.

The  3-Partition  problem can be reduced in polynomial time to the problem of finding the aggregation $q^{*}\in {\mathcal{P}}_m$ of some $p\in {\mathcal{P}}_n$, for which

$$D(q^{*}\parallel p)=\underset{q\in {\mathcal{A}}_m\left(p\right)}{min}D(q\parallel p).$$

Proof. 

The idea behind the following reduction can be summarized as follows: given an instance of 3-Partition, we transform it into a probability distribution p such that the lower bound $lb\left(p\right)$ is an aggregation of p if and only if the original instance of 3-Partition admits a solution. Let an arbitrary instance of 3-Partition be given, that is, let S be a multiset $\{a_1,\cdots ,a_n\}$ of $n=3m$ positive integers with ${\sum }_{i=1}^n{a}_i=mT$. Without loss of generality, we assume that the integers $a_i$ are ordered in a non-increasing fashion. We construct a valid instance p of our Problem 1 as follows. We set $p\in {\mathcal{P}}_{n+m}$ as follows:

$$p=\left(\underset{m \mathrm{times}}{\underbrace{\frac{1}{m+1},\cdots ,\frac{1}{m+1}}},\frac{a_1+2T}{(m+1)7mT},\cdots ,\frac{a_n+2T}{(m+1)7mT}\right).$$

(5)

Note that p is a probability distribution. In fact, since $n=3m$, we have

$$\sum _{i=1}^n\frac{a_i+2T}{(m+1)7mT}=\frac{1}{(m+1)7mT}\left(\sum _{i=1}^n(a_i+2T)\right)=\frac{7mT}{(m+1)7mT}=\frac{1}{m+1}.$$

Moreover, from (4) and (5), the probability distribution $lb\left(p\right)\in {\mathcal{P}}_m$ associated to p is as follows:

$$lb\left(p\right)=\left(\frac{p_1}{{\sum }_{j=1}^m{p}_j},\cdots ,\frac{p_m}{{\sum }_{j=1}^m{p}_j}\right)=\left(\frac{1}{m},\cdots ,\frac{1}{m}\right).$$

(6)

To prove the theorem, we show that the starting instance of 3-Partition is a Yes instance if and only if it holds that

$$\underset{q\in {\mathcal{A}}_m\left(p\right)}{min}D(q\parallel p)=log\frac{m+1}{m},$$

(7)

where p is given in (5).

We begin by assuming the given instance of 3-Partition is a Yes instance, that is, there is a partition of S into triplets $S_1,\cdots ,S_m$ such that

$$\sum _{a_i\in S_j}{a}_i=T,\forall j\in \{1,\cdots ,m\},$$

(8)

and we show that ${min}_{q\in {\mathcal{A}}_m\left(p\right)}D(q\parallel p)=log\frac{m+1}{m}$. By Lemma 1, (5), and equality (6), we have

$$\underset{q\in {\mathcal{A}}_m\left(p\right)}{min}D(q\parallel p)\ge D(lb\left(p\right)\parallel p)=\sum _{i=1}^m\frac{1}{m}log\frac{1/m}{1/(m+1)}=log\frac{m+1}{m}.$$

(9)

From (8), we have

$$\begin{array}{cc}\hfill \sum _{a_i\in S_j}\frac{a_i+2T}{(m+1)7mT}& =\frac{T}{(m+1)7mT}+\sum _{a_i\in S_j}\frac{2T}{(m+1)7mT}\hfill \\ \hfill & =\frac{T}{(m+1)7mT}+\frac{6T}{(m+1)7mT}\hfill \\ \hfill & =\frac{1}{(m+1)m},\forall j\in \{1,\cdots ,m\}.\hfill \end{array}$$

(10)

Let us define $q^{\prime }\in {\mathcal{P}}_m$ as follows:

$$q^{\prime }=\left(\frac{1}{m+1}+\sum _{a_i\in S_1}\frac{a_i+2T}{(m+1)7mT},\cdots ,\frac{1}{m+1}+\sum _{a_i\in S_m}\frac{a_i+2T}{(m+1)7mT}\right),$$

(11)

where, by (10),

$$\sum _{a_i\in S_j}\frac{a_i+2T}{(m+1)7mT}=\frac{1}{(m+1)m},\forall j\in \{1,\cdots ,m\}.$$

(12)

From (12) and from the fact that $S_1,\dots ,S_m$ are a partition of $\{a_1,\dots ,a_n\}$, we obtain $q^{\prime }\in {\mathcal{A}}_m\left(p\right)$, that is, $q^{\prime }$ is a valid aggregation of p (cfr., (5)). Moreover,

$$q^{\prime }=\left(\frac{1}{m},\cdots ,\frac{1}{m}\right),$$

and $D(q^{\prime }\parallel p)=log\frac{m+1}{m}$. Therefore, by (9) and that $q^{\prime }\in {\mathcal{A}}_m\left(p\right)$, we obtain

$$\underset{q\in {\mathcal{A}}_m\left(p\right)}{min}D(q\parallel p)=log\frac{m+1}{m},$$

as required.

To prove the opposite implication, we assume that p (as given in (5)) is a Yes instance, that is,

$$\underset{q\in {\mathcal{A}}_m\left(p\right)}{min}D(q\parallel p)=log\frac{m+1}{m}.$$

(13)

We show that the original instance of 3-Partition is also a Yes instance, that is, there is a partition of S into triplets $S_1,\cdots ,S_m$ such that

$$\sum _{a_i\in S_j}{a}_i=T,\forall j\in \{1,\cdots ,m\}.$$

(14)

Let $q^{*}$ be the element in ${\mathcal{A}}_m\left(p\right)$ that achieves the minimum in (13). Consequently, we have

$$\begin{array}{cc}\hfill log\frac{m+1}{m}& =D(q^{*}\parallel p)=\sum _{i=1}^m{q}_i^{*}log\frac{q_i^{*}}{p_i}=\sum _{i=1}^m{q}_i^{*}log\frac{1}{p_i}-H\left(q^{*}\right)\hfill \\ \hfill & =log(m+1)-H\left(q^{*}\right)(\mathrm{from}\left(5\right)),\hfill \end{array}$$

(15)

where $H\left(q^{*}\right)=-{\sum }_{i=1}^m{q}_i^{*}log{q}_i^{*}$ is the Shannon entropy of $q^{*}$. From (15), we obtain that $H\left(q^{*}\right)=logm$; hence, $q^{*}=(1/m,\dots ,1/m)$ (see [30], Thm. 2.6.4). Recalling that $q^{*}\in {\mathcal{A}}_m\left(p\right)$, we obtain that the uniform distribution

$$\left(\frac{1}{m},\cdots ,\frac{1}{m}\right)$$

(16)

is an aggregation of p. We note that the first m components of p, as defined in (5), cannot be aggregated among them to obtain (16), because $2/(m+1)>1/m$, for $m>2$. Therefore, in order to obtain (16) as an aggregation of p, there must exist a partition $S_1,\cdots ,S_m$ of $S=\{a_1,\dots ,a_n\}$ for which

$$\frac{1}{m+1}+\sum _{a_i\in S_j}\frac{a_i+2T}{(m+1)7mT}=\frac{1}{m},\forall j\in \{1,\cdots ,m\}.$$

(17)

From (17), we obtain

$$\sum _{a_i\in S_j}\frac{a_i+2T}{(m+1)7mT}=\frac{1}{m(m+1)},\forall j\in \{1,\cdots ,m\}.$$

(18)

From this, it follows that

$$2T|S_j|+\sum _{a_i\in S_j}{a}_i=7T,\forall j\in \{1,\cdots ,m\}.$$

(19)

We note that, for (19) to be true, there cannot exist any $S_j$ for which $|S_j|\ne 3$. Indeed, if there were a subset $S_j$ for which $|S_j|\ne 3$, there would be at least a subset $S_k$ for which $|S_k|>3$. Thus, for such an $S_k$, we would have

$$2T|S_k|+\sum _{a_i\in S_k}{a}_i\ge 8T+\sum _{a_i\in S_k}{a}_i>7T,$$

contradicting (19). Therefore, it holds that

$$|S_j|=3,\forall j\in \{1,\cdots ,m\}.$$

(20)

Moreover, from (19) and (20), we obtain

$$\sum _{a_i\in S_j}{a}_i=7T-2T\left|S_j\right|=T,\forall j\in \{1,\cdots ,m\}.$$

(21)

Thus, from (21), it follows that the subsets $S_1,\dots ,S_m$ give a partition of S into triplets, such that

$$\sum _{a_i\in S_j}{a}_i=T,\forall j\in \{1,\cdots ,m\}.$$

Therefore, the starting instance of 3-Partition is a Yes instance.    □

4. Approximation

Given $p\in {\mathcal{P}}_n$ and $m<n$, let $OPT$ denote the optimal value of the optimization problem (2), that is

$$OPT=\underset{q\in {\mathcal{A}}_m\left(p\right)}{min}D(q\parallel p).$$

(22)

In this section, we design a greedy algorithm to compute an aggregation $\overline{q}\in {\mathcal{A}}_m\left(p\right)$ of p such that

$$D(\overline{q}\parallel p)<OPT+1.$$

(23)

The idea behind our algorithm is to see the problem of computing an aggregation $q\in {\mathcal{A}}_m\left(p\right)$ as a bin packing problem with “overstuffing” (see [31] and references therein quoted), which is a bin packing where overfilling of bins is possible. In the classical bin packing problem, one is given a set of items, with their associated weights, and a set of bins with their associated capacities (usually, equal for all bins). The objective is to place all the items in the bins, trying to minimize a given cost function.

In our case, we have n items (corresponding to the components of p) with weights $p_1,\cdots ,p_n$, respectively, and m bins, corresponding to the components of $lb\left(p\right)$ (as defined in (4)) with capacities $lb{\left(p\right)}_1,\cdots ,lb{\left(p\right)}_m$. Our objective is to place all the n components of p into the m bins without exceeding the capacity $lb{\left(p\right)}_j$ of each bin j, $j=1,\dots ,m$, by more than $\left({\sum }_{i=1}^m{p}_i\right)lb{\left(p\right)}_j$. For such a purpose, the idea behind Algorithm 1 is quite straightforward. It behaves like a classical First-Fit bin packing: to place the ith item, it chooses the first bin j in which the item can be inserted without exceeding its capacity by more than $\left({\sum }_{i=1}^m{p}_i\right)lb{\left(p\right)}_j$. In the following, we will show that such a bin always exists and that fulfilling this objective is sufficient to ensure the approximation guarantee (23) we are seeking.

Algorithm 1: GreedyApprox

1. Compute $lb\left(p\right)=(p_1/{\sum }_{j=1}^m{p}_j,\cdots ,p_m/{\sum }_{j=1}^m{p}_j)$; 2. Let $l{b}_j^i$ be the content of bin j after the first i components of p have been placed ($l{b}_j^0=0$ for each $j\in \{1,\cdots ,m\}$); 3. For $i=0,\cdots ,n-1$         Let j be the smallest bin index for which holds that         $l{b}_j^i+p_{i+1}<(1+{\sum }_{j=1}^m{p}_j)lb{\left(p\right)}_j$, place $p_{i+1}$ into the j-th bin:             $l{b}_j^{i+1}=l{b}_j^i+p_{i+1},$             $l{b}_k^{i+1}=l{b}_k^i$, for each $k\ne j$; 4. Output $\overline{q}=(l{b}_1^n,\dots ,l{b}_m^n)$.

The step 3 of GreedyApprox operates as in the classical First-Fit bin packing algorithm. Therefore, it can be implemented to run in $O(nlogm)$ time, as discussed in [32]. In fact, each iteration of the loop in step 3 can be implemented in $O(logm)$-time by using a balanced binary search tree with height $O(logm)$ that has a leaf for each bin and in which each node keeps track of the largest remaining capacity of all the bins in its subtree.

Lemma 2.

GreedyApprox computes a valid aggregation $\overline{q}\in {\mathcal{A}}_m\left(p\right)$ of $p\in {\mathcal{P}}_n$. Moreover, it holds that

$$D(\overline{q}\parallel lb\left(p\right))<log\left(1+\sum _{j=1}^m{p}_j\right).$$

(24)

Proof. 

We first prove that each component $p_i$ of p is placed in some bin. This implies that $\overline{q}\in {\mathcal{A}}_m\left(p\right)$.

For each step $i=0,\cdots ,m-1$, there is always a bin in which the algorithm places $p_{i+1}$. In fact, the capacity $lb{\left(p\right)}_j$ of bin j satisfies the relation:

$$lb{\left(p\right)}_j=\frac{p_j}{{\sum }_{\ell =1}^m{p}_{\ell }}>p_j,\forall j\in \{1,\cdots ,m\}.$$

Let us consider an arbitrary step $m\le i<n$, in which the algorithm has placed the first i components of p and needs to place $p_{i+1}$ into some bin. We show that, in this case also, there is always a bin j in which the algorithm places the item $p_{i+1}$, without exceeding the capacity $lb{\left(p\right)}_j$ of the bin j by more than $\left({\sum }_{\ell =1}^m{p}_{\ell }\right)lb{\left(p\right)}_j$.

First, notice that in each step i, $m\le i<n$, there is at least a bin k whose content $l{b}_k^i$does not exceed its capacity $lb{\left(p\right)}_k$; that is, for which $l{b}_k^i<lb{\left(p\right)}_k$ holds. Were this the opposite, for all bins j, we would have $l{b}_j^i\ge lb{\left(p\right)}_j$; then, we would also have

$$\sum _{j=1}^{m}l{b}_j^i\ge \sum _{j=1}^{m}lb{\left(p\right)}_j=1.$$

(25)

However, this is not possible since we have placed only the first $i<n$ components of p, and therefore, it holds that

$$\sum _{j=1}^{m}l{b}_j^i=\sum _{j=1}^i{p}_j<\sum _{j=1}^n{p}_j=1,$$

contradicting (25). Consequently, let k be the smallest integer for which the content of the k-th bin does not exceed its capacity, i.e., for which $l{b}_k^i<lb{\left(p\right)}_k$. For such a bin k, we obtain

$$\begin{array}{cc}\hfill \left(1+\sum _{j=1}^m{p}_j\right)lb{\left(p\right)}_k& =lb{\left(p\right)}_k+\left(\sum _{j=1}^m{p}_j\right)lb{\left(p\right)}_k\hfill \\ \hfill & =lb{\left(p\right)}_k+\left(\sum _{j=1}^m{p}_j\right)\frac{p_k}{{\sum }_{j=1}^m{p}_j}\hfill \\ \hfill & =lb{\left(p\right)}_k+p_k\hfill \\ \hfill & >l{b}_k^i+p_k(\mathrm{since} lb{\left(p\right)}_k>l{b}_k^i)\hfill \\ \hfill & \ge l{b}_k^i+p_{i+1}(\mathrm{since} p_k\ge p_{i+1}).\hfill \end{array}$$

(26)

Thus, from (26), one derives that the algorithm places $p_{i+1}$ into the bin k without exceeding its capacity $lb{\left(p\right)}_k$ by more than $\left({\sum }_{j=1}^m{p}_j\right)lb{\left(p\right)}_k$.

The reasoning applies to each $i<n$, thus proving that GreedyApprox correctly assigns each component $p_i$ of p to a bin, effectively computing an aggregation of p. Moreover, from the instructions of step 3 of GreedyApprox, the output is an aggregation $\overline{q}=({\overline{q}}_1,\dots ,{\overline{q}}_m)\in {\mathcal{A}}_m\left(p\right)$, for which the following crucial relation holds:

$${\overline{q}}_i<\left(1+\sum _{j=1}^m{p}_j\right)lb{\left(p\right)}_i,\forall i\in \{1,\cdots ,m\}.$$

(27)

Let us now prove that $D(\overline{q}\parallel lb\left(p\right))<log\left(1+{\sum }_{j=1}^m{p}_j\right)$. We have

$$\begin{array}{cc}\hfill D(\overline{q}\parallel lb\left(p\right))& =\sum _{i=1}^m{\overline{q}}_{i}log\frac{{\overline{q}}_i}{lb{\left(p\right)}_i}\hfill \\ \hfill & <\sum _{i=1}^m{\overline{q}}_{i}log\frac{(1+{\sum }_{j=1}^m{p}_j)lb{\left(p\right)}_i}{lb{\left(p\right)}_i}(\mathrm{from} (27\left)\right)\hfill \\ \hfill & =log\left(1+\sum _{j=1}^m{p}_j\right).\hfill \end{array}$$

□

We need the following technical lemma to show the approximation guarantee of GreedyApprox.

Lemma 3.

Let $q\in {\mathcal{P}}_m$ and $p\in {\mathcal{P}}_n$ be two arbitrary probability distributions with $m<n$. It holds that

$$D(q\parallel p)=D(q\parallel lb(p\left)\right)+D\left(lb\right(p)\parallel p),$$

(28)

where $lb\left(p\right)=(lb{\left(p\right)}_1,\dots ,lb{\left(b\right)}_m)=(p_1/{\sum }_{i=1}^m{p}_i,\cdots ,p_m/{\sum }_{i=1}^m{p}_i)$.

Proof. 

$$\begin{array}{cc}\hfill D(q\parallel p)& =\sum _{i=1}^m{q}_{i}log\frac{q_i}{p_i}=\sum _{i=1}^m{q}_{i}log\frac{q_i}{p_i\frac{{\sum }_{j=1}^m{p}_j}{{\sum }_{j=1}^m{p}_j}}\hfill \\ \hfill & =\sum _{i=1}^m{q}_{i}log\frac{q_i}{\frac{p_i}{{\sum }_{j=1}^m{p}_j}}+\sum _{i=1}^m{q}_{i}log\frac{1}{{\sum }_{j=1}^m{p}_j}\hfill \\ \hfill & =\sum _{i=1}^m{q}_{i}log\frac{q_i}{lb{\left(p\right)}_i}+\sum _{i=1}^m{q}_{i}log\frac{1}{{\sum }_{j=1}^m{p}_j}(\mathrm{since} lb{\left(p\right)}_i=p_i/{\sum }_{j=1}^m{p}_j)\hfill \\ \hfill & =D(q\parallel lb\left(p\right))+log\frac{1}{{\sum }_{j=1}^m{p}_j}=D(q\parallel lb\left(p\right))+D(lb\left(p\right)\parallel p).\hfill \end{array}$$

□

The following theorem is the main result of this section.

Theorem 2.

For any $p\in {\mathcal{P}}_n$ and $m<n$, GreedyApprox produces an aggregation $\overline{q}\in {\mathcal{A}}_m\left(p\right)$ of p such that

$$D(\overline{q}\parallel p)<OPT+1,$$

(29)

where $OPT={min}_{q\in {\mathcal{A}}_m\left(p\right)}D(q\parallel p).$

Proof. 

From Lemma 3, we have

$$D(\overline{q}\parallel p)=D(\overline{q}\parallel lb\left(p\right))+D(lb\left(p\right)\parallel p),$$

(30)

and from Theorem 2, we know that the produced aggregation $\overline{q}$ of p satisfies the relation

$$D(\overline{q}\parallel lb\left(p\right))<log\left(1+\sum _{j=1}^m{p}_j\right).$$

(31)

Putting it all together, we obtain:

$$\begin{array}{cc}\hfill D(\overline{q}\parallel p)& =D(\overline{q}\parallel lb\left(p\right))+D(lb\left(p\right)\parallel p)\hfill \\ \hfill & <log\left(1+\sum _{j=1}^m{p}_j\right)+D(lb\left(p\right)\parallel p)(\mathrm{from} (31\left)\right)\hfill \\ \hfill & =log\left(1+\sum _{j=1}^m{p}_j\right)-log\left(\sum _{j=1}^m{p}_j\right)\hfill \\ \hfill & <-log\left(\sum _{j=1}^m{p}_j\right)+1(\mathrm{since} 1+{\sum }_{j=1}^m{p}_j<2)\hfill \\ \hfill & \le OPT+1(\mathrm{from} \mathrm{Lemma} 1).\hfill \end{array}$$

□

5. Concluding Remarks

In this paper, we examined the problem of approximating n-dimensional probability distributions with m-dimensional ones using the Kullback–Leibler divergence as the measure of closeness. We demonstrated that this problem is strongly NP-hard and introduced an approximation algorithm for solving the problem with guaranteed performance.

Moreover, we conclude by pointing out that the analysis of GreedyApprox presented in Theorem 2 is tight. Let $p\in {\mathcal{P}}_3$ be

$$p=\left(\frac{1}{2}-ϵ,\frac{1}{2}-ϵ,2ϵ\right),$$

where $ϵ>0$. The application of GreedyApprox on p produces the aggregation $\overline{q}\in {\mathcal{P}}_2$ given by

$$\overline{q}=(1-2ϵ,2ϵ),$$

whereas one can see that the optimal aggregation $q^{*}\in {\mathcal{P}}_2$ is equal to

$$q^{*}=\left(\frac{1}{2}+ϵ,\frac{1}{2}-ϵ\right).$$

Hence, for $ϵ\to 0$, we have

$$D(\overline{q}\parallel p)=(1-2ϵ)log\frac{1-2ϵ}{\frac{1}{2}-ϵ}+2ϵlog\frac{2ϵ}{\frac{1}{2}-ϵ}\to 1,$$

while

$$OPT=D(q^{*}\parallel p)=\left(\frac{1}{2}+ϵ\right)log\frac{\frac{1}{2}+ϵ}{\frac{1}{2}-ϵ}+\left(\frac{1}{2}-ϵ\right)log\frac{\frac{1}{2}-ϵ}{\frac{1}{2}-ϵ}\to 0.$$

Therefore, to improve our approximation guarantee, one should use a bin packing heuristic different from the First-Fit as employed in GreedyApprox. Another interesting open problem is to provide an approximation algorithm with a (small) multiplicative approximation guarantee. However, both problems mentioned above would probably require a different approach, and we leave that to future investigations.

Another interesting line of research would be to extend our findings to different divergence measures (e.g., [33] and references quoted therein).