Rate-Distortion Bounds for Kernel-Based Distortion Measures ^†

Abstract

Kernel methods have been used for turning linear learning algorithms into nonlinear ones. These nonlinear algorithms measure distances between data points by the distance in the kernel-induced feature space. In lossy data compression, the optimal tradeoff between the number of quantized points and the incurred distortion is characterized by the rate-distortion function. However, the rate-distortion functions associated with distortion measures involving kernel feature mapping have yet to be analyzed. We consider two reconstruction schemes, reconstruction in input space and reconstruction in feature space, and provide bounds to the rate-distortion functions for these schemes. Comparison of the derived bounds to the quantizer performance obtained by the kernel $\mathtt{K}$-means method suggests that the rate-distortion bounds for input space and feature space reconstructions are informative at low and high distortion levels, respectively.

1. Introduction

Kernel methods have been widely used for nonlinear learning problems combined with linear learning algorithms such as the support vector machine and the principal component analysis [1]. By the so-called kernel trick, kernel-based methods can use linear learning methods in the kernel-induced feature space without explicitly computing the high-dimensional feature mapping. Kernel-based methods measure the dissimilarity between data points by the distance in the feature space, which, in input space, corresponds to a distance measure involving the feature mapping [2]. If a kernel-based learning method is used as a lossy source coding scheme, its optimal rate-distortion tradeoff is indicated by the rate-distortion function associated with the distortion measure defined by the kernel feature map [3]. Successful applications of kernel methods in learning problems and flexibility to create various distance measures suggest that kernel-based distortion measures can be suitable for certain lossy compression problems. However, the rate-distortion function of such a distortion measure has yet to be evaluated analytically. Although there are several kernel-based approaches to vector quantization [4,5], their rate-distortion tradeoffs are still unknown.

In this paper, we derive bounds for the rate-distortion functions for kernel-based distortion measures. We consider two schemes to reconstruct inputs in lossy coding methods. One is to obtain a reconstruction in the original input space. Since kernel methods usually yield results of learning by the linear combination of vectors in feature space, we need an additional step to obtain the reconstruction in input space, such as preimaging [6]. The other is to consider the linear combination of feature vectors as the reconstruction and measure the distortion in the feature space directly. We formulate the two reconstruction schemes (Section 3.1 and Section 3.2), and prove that the rate-distortion function of input space reconstruction provides an upper bound of that of feature space reconstruction (Section 3.3). We derive lower and upper bounds to the rate-distortion function of input space reconstruction, which are computable only by $one$-dimensional numerical integrations in the case of translation invariant and isotropic kernel functions (Section 4.1 and Section 4.2). We also provide an upper bound to the rate-distortion function of feature space reconstruction for general positive definite kernel functions (Section 4.4). In the usual applications of kernel-based quantization algorithms, one fixes the rate by determining the number of quantized points, and minimizes the average distortion for training data. The distortion-rate function, which is the inverse function of the rate-distortion function, shows the minimum achievable expected distortion (or distortion for test data) at the fixed rate. The derived bounds approximately characterize such optimal tradeoffs between the rate and expected distortion.

Furthermore, we design a vector quantizer using the kernel $\mathtt{K}$-means method and compare its performance with the derived rate-distortion bounds (Section 5). We also compute the preimages of the quantized points in feature space to investigate the performance of the quantizer in input space. It is suggested through the experiments using synthetic and image data that the rate-distortion bounds of reconstruction in input space are accurate at low distortion levels while the upper bound for reconstruction in feature space is informative at high distortion levels.

2. Rate-Distortion Function

Let X and Y be random variables of input and reconstruction taking values in $\mathcal{X}$ and $\mathcal{Y}$, respectively. For the non-negative distortion measure between x and y, $d(x,y)$, the rate-distortion function $R\left(D\right)$ of the source $X\sim p\left(x\right)$ is defined by

$$R\left(D\right)=\underset{q\left(y\right|x):E[d(X,Y)]\le D}{inf}I\left(q\right),$$

(1)

where $I\left(q\right)=I(X;Y)$ is the mutual information and E denotes the expectation with respect to $q\left(y\right|x\left)p\right(x)$. $R\left(D\right)$ shows the minimum achievable rate R under the given distortion measure d [3,7]. The distortion-rate function is the inverse function of the rate-distortion function and denoted by $D\left(R\right)$.

If the conditional distributions $q_s\left(y\right|x)$ achieve the minimum of the following Lagrange functional parameterized by $s\ge 0$,

$$L\left(q\right)=I\left(q\right)+s\left(E\left[d\right(X,Y\left)\right]-D\right),$$

then, the rate-distortion function is parametrically given by

$$\begin{array}{ccc}\hfill R\left(D_s\right)& =& I\left(q_s\right),\hfill \\ \hfill D_s& =& \int q_s\left(y\right|x)p\left(x\right)d(x,y)dxdy.\hfill \end{array}$$

The parameter s corresponds to the (negated) slope of the tangent of $R\left(D\right)$ at $(D_s,R\left(D_s\right))$ and hence is referred to as the slope parameter [3]. Alternatively, if there exists a marginal reconstruction density $q_s\left(y\right)$ that minimizes the functional,

$$F\left(q\right)=-\frac{1}{s}E\left[log\int e^{-sd(X,y)}q\left(y\right)dy\right],$$

then the optimal conditional reconstruction distributions are given by

$$q_s\left(y\right|x)=\frac{e^{-sd(x,y)}{q}_s\left(y\right)}{\int e^{-sd(x,y)}{q}_s\left(y\right)dy}$$

(2)

(see, for example, [3,8]).

From the properties of the rate-distortion function $R\left(D\right)$, we know that $R\left(D\right)>0$ for $0<D<D_{max}$, where

$$D_{max}=\underset{y}{inf}\int p\left(x\right)d(x,y)dx,$$

(3)

and $R\left(D\right)=0$ for $D\ge D_{max}$ [3] (p. 90). Hence, $D_{max}={lim}_{R\to 0}D\left(R\right)$.

3. Kernel-Based Distortion Measures

In kernel-based learning methods, data points in input space $\mathcal{X}$ are mapped into some high-dimensional feature space H by a feature mapping $\varphi$. Then, the similarity between the two points x and y in $\mathcal{X}$ is measured by the inner product $⟨\varphi \left(x\right),\varphi \left(y\right)⟩$ in H.

The inner product is directly evaluated by a nonlinear function in input space

$$K(x,y)=⟨\varphi \left(x\right),\varphi \left(y\right)⟩,$$

(4)

which is called the kernel function. Mercer’s theorem ensures that there exists some $\varphi$ such that Equation (4) holds if K is a positive definite kernel [1]. This enables us to avoid explicitly computing the feature map $\varphi$ in the potentially high-dimensional space H, which is called the kernel trick. A lot of learning methods that can be expressed by only the inner products between data points have been kernelized [1].

We identify the feature space H with the reproducing kernel Hilbert space (RKHS) associated with the kernel function K by the canonical feature map, $\varphi \left(x\right)=K(·,x)$ [9] (Lemma 4.19). We assume that the input space $\mathcal{X}$ is a subset of ${\mathbb{R}}^m$, and the kernel function K is continuous [9] (Lemma 4.29). We focus on the squared norm in feature space as the distortion measure, and consider two reconstruction schemes in the following respective subsections.

3.1. Reconstruction in Input Space

If we restrict ourselves to the reconstruction in input space, that is, the reconstruction $y\in \mathcal{X}\subset {\mathbb{R}}^m$ is computed for each input $x\in \mathcal{X}$, the distortion measure is naturally defined by

$$\begin{array}{ccc}\hfill d_{\mathrm{inp}}(x,y)& =& \left|\right|\varphi \left(x\right)-{\varphi \left(y\right)\left|\right|}^2\hfill \\ \hfill & =& K(x,x)+K(y,y)-2K(x,y).\hfill \end{array}$$

(5)

Note that the reconstruction $\varphi \left(y\right)$ of $\varphi \left(x\right)$ is restricted to the subset of the feature space, $\left\{\varphi \right(y);y\in \mathcal{X}\}$. To obtain a reconstruction in input space, we need a technique such as preimaging [6].

This is a difference distortion measure if and only if the kernel function is translation invariant, that is, $K(x+a,y+a)=K(x,y)$ for any $a\in \mathcal{X}$. In this case, the distortion measure is expressed as

$$d_{\mathrm{inp}}(x,y)=\rho (x-y),$$

(6)

where $\rho \left(z\right)=2(C-K(z,0\left)\right)$ and $C=K(0,0)$. The rate-distortion function (distortion-rate function, resp.) for this distortion measure is denoted by $R_{\mathrm{inp}}\left(D\right)$ ($D_{\mathrm{inp}}\left(R\right)$, resp.) and the maximum distortion $D_{max}$ in Equation (3) is denoted by $D_{max,\mathrm{inp}}$, that is,

$$D_{max,\mathrm{inp}}=E\left[K(X,X)\right]+\underset{y}{inf}\left\{K(y,y)-2E\left[K\right(X,y\left)\right]\right\},$$

(7)

which is in the translation invariant case, $D_{max,\mathrm{inp}}=2\left(C-{sup}_{y}E\left[K(X,y)\right]\right)$.

3.2. Reconstruction in Feature Space

Suppose we have a sample of length n in input space, $S=\{x_1,\dots ,x_n\}$ so that $\{\varphi \left(x_1\right),\dots ,\varphi \left(x_n\right)\}$ spans a linear subspace in feature space. If we compute the reconstruction by the linear combination ${\sum }_{i=1}^n{\alpha }_i\varphi \left(x_i\right)$ for ${\alpha }_i\in \mathbb{R},i=1,\dots ,n$, and consider it as the reconstruction in feature space, the distortion can be measured by

$$\begin{array}{ccc}\hfill d_{\mathrm{fea}}(x,\mathit{\alpha })& =& d_{\mathrm{fea}}^{\left[S\right]}(x,\mathit{\alpha })={∥\varphi \left(x\right)-\sum _{i=1}^n{\alpha }_i\varphi \left(x_i\right)∥}^2\hfill \\ \hfill & =& K(x,x)-2{\mathit{\alpha }}^T\mathit{k}\left(x\right)+{\mathit{\alpha }}^T\mathit{K}\mathit{\alpha },\hfill \end{array}$$

(8)

where $\mathit{\alpha }={({\alpha }_1,\dots ,{\alpha }_n)}^T\in {\mathbb{R}}^n$,

$$\mathit{k}\left(x\right)={(K(x_1,x),\dots ,K(x_n,x))}^T,$$

and $\mathit{K}={\left(K(x_i,x_j)\right)}_{ij}$ is the Gram matrix. Note that the reconstruction is identified with the coefficients $\mathit{\alpha }$ whose domain is not identical to the input space $\mathcal{X}$. Although the distortion measure $d_{\mathrm{fea}}$ depends on the sample S, we omit the dependence in the notation since we consider a fixed design of S for a sufficiently large n. The sample does not have to be distributed according to the source distribution, while it is required to overspread the support of the source.

The rate-distortion function (distortion-rate function, resp.) for this distortion measure is denoted by $R_{\mathrm{fea}}\left(D\right)$ ($D_{\mathrm{fea}}\left(R\right)$, resp.) and the maximum distortion $D_{max}$ in Equation (3) is given by

$$D_{max,\mathrm{fea}}=E\left[K(X,X)\right]-E{\left[\mathit{k}\left(X\right)\right]}^T{\mathit{K}}^{-1}E\left[\mathit{k}\left(X\right)\right],$$

(9)

which is derived from the direct minimization of the quadratic function of $\mathit{\alpha }$, $\int d_{\mathrm{fea}}(x,\mathit{\alpha })p\left(x\right)dx$.

3.3. $R_{\mathrm{inp}}\left(D\right)$ and $R_{\mathrm{fea}}\left(D\right)$

The following theorem claims that $R_{\mathrm{inp}}\left(D\right)$ provides an upper bound of $R_{\mathrm{fea}}\left(D\right)$ when n is sufficiently large.

Theorem 1.

If the input space $\mathcal{X}$ is bounded, and there exists a conditional density achieving the infimum in the definition of $R_{\mathrm{inp}}\left(D\right)$, for any $\epsilon >0$, $D\ge \epsilon$, and sufficiently large n, the following inequality holds:

$$R_{\mathrm{fea}}(D+\epsilon )\le R_{\mathrm{inp}}\left(D\right).$$

The proof is given in Appendix A. This theorem shows that the feature space reconstruction gives better rates since a single feature vector $\varphi \left(y\right)$ can be approximated by a linear combination ${\sum }_{i=1}^n{\alpha }_i\varphi \left(x_i\right)$ when n is sufficiently large.

4. Rate-Distortion Bounds

Since the rate-distortion problem (Section 2) is rarely solved in a closed form [8], we derive bounds to $R_{\mathrm{inp}}\left(D\right)$ and $R_{\mathrm{fea}}\left(D\right)$.

4.1. Lower Bound to $R_{\mathrm{inp}}\left(D\right)$

Although the Shannon lower bound to $R\left(D\right)$ is defined for difference distortion measures in general [3] (p. 92), it diverges to $-\infty$ for the distortion measure in Equation (6) since $\int e^{-s\rho \left(z\right)}dz$ diverges to ∞. Hence, we consider an improved lower bound, which was introduced by [3] (p. 140). Let $Q_B$ be the probability that $\parallel X\parallel \le B$. Then, $R\left(D\right)$ is lower-bounded as

$$R\left(D\right)\ge Q_B\left\{h\left(p_B\right)-\underset{g\in G_{B,D}}{max}h\left(g\right)\right\},$$

(10)

where h denotes the differential entropy,

$$p_B\left(x\right)=\frac{1}{Q_B}p\left(x\right)u(B-\parallel x\parallel ),$$

(11)

and u is the step function. $G_{B,D}$ is the set of all probability densities $g(·)$ for which $g\left(x\right)=0$ for $\parallel x\parallel >B$ and $\int \rho \left(z\right)g\left(z\right)dz\le D/Q_B$.

In the case of the distortion measure in Equation (6), the maximum in Equation (10) is explicitly given by

$$g_s\left(z\right)=\frac{1}{C_{B,s}}exp\left(2sK(z,0)\right)u(B-\parallel z\parallel ),$$

(12)

where $C_{B,s}={\int }_{\parallel z\parallel \le B}{e}^{2sK(z,0)}dz$ for s related to D by $\int \rho \left(z\right)g_s\left(z\right)dz=D/Q_B$. Since its differential entropy is

$$h\left(g_s\right)=-s\frac{\partial log{C}_{B,s}}{\partial s}+log{C}_{B,s},$$

(13)

we arrive at the following theorem.

Theorem 2.

The rate distortion function $R_{\mathrm{inp}}\left(D\right)$ is parametrically lower-bounded as

$$\begin{array}{ccc}\hfill R_{\mathrm{inp}}\left(D_s\right)\ge R_{\mathrm{inp},L}\left(D_s\right)& =& Q_B\left\{h\left(p_B\right)+s\frac{\partial log{C}_{B,s}}{\partial s}-log{C}_{B,s}\right\},\hfill \\ \hfill D_s& =& Q_B\left\{2C-\frac{\partial log{C}_{B,s}}{\partial s}\right\}.\hfill \end{array}$$

(14)

If we further assume that the kernel function is radial, that is, $K(x,y)=K(x-y,0)=k(\parallel x-y\parallel )$ for some function k, the integrations above reduce to $one$-dimensional ones,

$$C_{B,s}=A\left(m\right){\int }_0^B{r}^{m-1}{e}^{2sk\left(r\right)}dr,$$

and

$$\begin{array}{ccc}\hfill \frac{\partial log{C}_{B,s}}{\partial s}& =& 2{\int }_{\parallel z\parallel \le B}K(z,0)e^{2sK(z,0)}dz\hfill \\ \hfill & =& 2A\left(m\right){\int }_0^B{r}^{m-1}k\left(r\right)e^{2sk\left(r\right)}dr,\hfill \end{array}$$

(15)

where $A\left(m\right)=\frac{m{\sqrt{\pi }}^m}{\mathrm{\Gamma }(m/2+1)}$ is the area of the m-dimensional unit sphere, and $\mathrm{\Gamma }$ is the gamma function.

4.2. Upper Bound to $R_{\mathrm{inp}}\left(D\right)$

If $d_{\mathrm{inp}}$ in Equation (5) is a difference distortion measure, that is, K is translation invariant, by choosing $q\left(y\right|x)=g_s(y-x)$ for the density $g_s$ in Equation (12), the following upper bound is obtained,

$$\begin{array}{ccc}\hfill R_{\mathrm{inp}}\left(D_s\right)\le R_{\mathrm{inp},U}\left(D_s\right)& =& h(g_s\ast p)-h\left(g_s\right)\hfill \end{array}$$

(16)

$$\begin{array}{cccc}\hfill & \hfill D_s& =& 2C-\frac{\partial log{C}_{B,s}}{\partial s},\hfill \end{array}$$

(17)

where $h\left(g_s\right)$ is given by Equation (13) and $(g_s\ast p)\left(y\right)=\int g_s(y-x)p\left(x\right)dx$ is the convolution between $g_s$ and p. This type of upper bound was used to prove the asymptotic tightness of the Shannon lower bound (as $D\to 0$) for a class of general sources and distortion measures [3,10,11,12]. However, this upper bound requires the evaluation of the differential entropy of the convolution.

The following theorem is derived from the facts that the spherical Gaussian distribution maximizes the entropy under the constraint that $E[\parallel X{\parallel }^2]$ is no greater than a constant, and that ${E[\parallel Y\parallel }^2{]=E[\parallel X\parallel }^2{]+E[\parallel Z\parallel }^2]$ holds for $Y=X+Z\sim g_s\ast p$.

Theorem 3.

If the kernel function is translation invariant and radial, $K(x,y)=k(\parallel x-y\parallel )$, then $R_{\mathrm{inp}}\left(D\right)$ is parametrically upper-bounded as

$$R_{\mathrm{inp}}\left(D_s\right)\le R_{\mathrm{inp},G}\left(D_s\right)=\frac{m}{2}log\left(2\pi e(v_p+v_s)\right)-h\left(g_s\right),$$

where

$$\begin{array}{ccc}\hfill v_p& =& \frac{1}{m}\int {\parallel x-\mu \parallel }^{2}p\left(x\right)dx,\hfill \\ \hfill \mu & =& \int xp\left(x\right)dx,\hfill \\ \hfill v_s& =& \frac{1}{m}\int {\parallel x\parallel }^2{g}_s\left(x\right)dx\hfill \\ \hfill & =& \frac{A\left(m\right)}{m{C}_{B,s}}{\int }_0^B{r}^{m+1}{e}^{2sk\left(r\right)}dr,\hfill \end{array}$$

(18)

and $D_s$ is given by Equation (17) (and Equation (15)).

4.3. Rate-Distortion Dimension

In this section, we evaluate the rate-distortion dimension [13] of the kernel-based distortion measure in Equation (5) to investigate its property. We focus on the radial kernel, $K(x,y)=k(\parallel x-y\parallel )$, also in this section, and assume that

$$\underset{r\to 0}{lim}\frac{k\left(r\right)-k\left(0\right)}{r^{\alpha }}=-\beta$$

(19)

holds for some $\alpha >0$ and $\beta >0$. For example, the Gaussian kernel, $k\left(r\right)=exp\left(-\gamma r^2\right)(\gamma >0)$, satisfies Equation (19) for $\alpha =2$ and $\beta =\gamma$.

To examine the limit $D\to 0$ of $R_{\mathrm{inp}}\left(D\right)$, we consider the asymptotic case of $s\to \infty$. Since $k\left(r\right)=k\left(0\right)-\beta r^{\alpha }+o\left(r^{\alpha }\right)$, it follows that

$$\begin{array}{ccc}\hfill C_{B,s}& =& A\left(m\right){\int }_0^B{e}^{2sk\left(r\right)r^{m-1}}dr\hfill \\ \hfill & =& A\left(m\right)e^{2sk\left(0\right)}\frac{1}{\alpha }{\left(\frac{1}{s\beta }\right)}^{m/\alpha }\left\{\mathrm{\Gamma }\left(\frac{m}{\alpha }\right)+o\left(1\right)\right\},\hfill \end{array}$$

$${\int }_0^{B}2k\left(r\right)e^{2sk\left(r\right)r^{m-1}}dr=2k\left(0\right)\frac{C_{B,s}}{A\left(m\right)}-2e^{2sk\left(0\right)}\frac{1}{s\alpha }{\left(\frac{1}{s\beta }\right)}^{1+m/\alpha }\left\{\mathrm{\Gamma }\left(1+\frac{m}{\alpha }\right)+o\left(1\right)\right\},$$

and

$$\begin{array}{ccc}\hfill \frac{\partial log{C}_{B,s}}{\partial s}& =& \frac{{\int }_0^{B}2k\left(r\right)e^{2sk\left(r\right)r^{m-1}}dr}{{\int }_0^B{e}^{2sk\left(r\right)r^{m-1}}dr}\hfill \\ \hfill & =& 2k\left(0\right)-\frac{m}{s\alpha \beta }+o\left(\frac{1}{s}\right).\hfill \end{array}$$

Thus, we have from Equations (14) and (17),

$$-log{D}_s=logs+O\left(1\right),$$

for both the lower and upper bounds, and from Equation (13),

$$\begin{array}{ccc}\hfill h\left(g_s\right)& =& -\frac{m}{\alpha }logs+O\left(1\right)\hfill \\ \hfill & =& \frac{m}{\alpha }log{D}_s+O\left(1\right).\hfill \end{array}$$

(20)

Since $d_{\mathrm{inp}}$ in Equation (5) is a norm squared for a valid RKHS kernel K, the rate-distortion dimension of the source distribution p is defined by [13],

$${\dim }_R\left(p\right)=\underset{D\to 0}{lim}\frac{R_{\mathrm{inp}}\left(D\right)}{-\frac{1}{2}logD}.$$

(21)

From Theorems 2 and 3 and Equation (20), we conclude the following.

Theorem 4.

If the source has a finite differential entropy, positive and finite $v_p$ defined in Equation (18), and a bounded support, that is, there exists a finite $B>0$ such that $Q_B=1$ in Equation (11), and the radial kernel, $K(x,y)=k(\parallel x-y\parallel )$ satisfies Equation (19) for $\alpha >0$ and $\beta >0$, then the rate-distortion dimension Equation (21) of $R_{\mathrm{inp}}\left(D\right)$ is given by

$${\dim }_R\left(p\right)=\frac{2m}{\alpha }.$$

(22)

This theorem shows that the rate-distortion dimension is dependent only on the dimensionality of the input space and independent of the dimensionality of the feature space. In the case of the linear kernel, $K(x,y)=⟨x,y⟩$, with $\varphi \left(x\right)=x$, the distortion measure in Equation (5) reduces to the usual squared distortion measure, ${\parallel x-y\parallel }^2$. It can be shown that under norm-based distortion measures including the squared distortion measure, the rate-distortion dimension of a source with an m-dimensional density is m [11,12]. From the preceding theorem, this is also the case for a general radial kernel if the kernel function has the order $\alpha =2$ as the Gaussian kernel. Expression (22) of the rate-distortion dimension will be examined through a numerical experiment in Section 5.1.

4.4. Upper Bound to $R_{\mathrm{fea}}\left(D\right)$

We construct an upper bound to the rate-distortion function $R_{\mathrm{fea}}\left(D\right)$. We choose the conditional distribution of the reconstruction by

$$q\left(\mathit{\alpha }\right|x)=N(\mathit{\alpha };{\mathit{m}}_K\left(x\right),{\tilde{\mathit{K}}}^{-1}/2s),$$

(23)

where $\tilde{\mathit{K}}=\mathit{K}+c\mathit{I}$,

$${\mathit{m}}_K\left(x\right)={\tilde{\mathit{K}}}^{-1}\mathit{k}\left(x\right),$$

and $N(·;\mathit{m},\Sigma )$ denotes the n-dimensional normal density with mean $\mathit{m}$ and covariance matrix $\Sigma$. Here, we have introduced the regularization constant $c\ge 0$ with the $n\times n$ identity matrix $\mathit{I}$. The conditional distribution in Equation (23) is implied by Equation (2) and the approximation $q_s\left(\mathit{\alpha }\right)=N(\mathit{\alpha };\mathbf{0},\mathit{I}/\left(2sc\right))$. This reconstruction distribution yields the following upper bound:

$$\begin{array}{ccc}\hfill R_{\mathrm{fea}}\left(D_s\right)\le R_{\mathrm{fea},U}\left(D_s\right)& =& h\left(M_p\right)-h\left(N(\mathit{\alpha };{\mathit{m}}_K\left(x\right),{\tilde{\mathit{K}}}^{-1}/2s)\right),\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill D_s& =& \frac{n-c\mathrm{tr}\left\{{\tilde{\mathit{K}}}^{-1}\right\}}{2s}+D_{min}\left(c\right),\hfill \end{array}$$

(25)

where $M_p\left(\mathit{\alpha }\right)=\int N(\mathit{\alpha };{\mathit{m}}_K\left(x\right),{\tilde{\mathit{K}}}^{-1}/2s)p\left(x\right)dx$,

$$h\left(N(\mathit{\alpha };{\mathit{m}}_K\left(x\right),{\tilde{\mathit{K}}}^{-1}/2s)\right)=\frac{n}{2}log\left(\frac{\pi e}{s}{\left|\tilde{\mathit{K}}\right|}^{1/n}\right),$$

(26)

which is independent of the input x, and

$$D_{min}\left(c\right)=E\left[K(X,X)\right]-\mathrm{tr}\left\{{\tilde{\mathit{K}}}^{-1}E\left[\mathit{k}\left(X\right)\mathit{k}{\left(X\right)}^T\right]\right\}-c\mathrm{tr}\left\{{\tilde{\mathit{K}}}^{-1}E\left[\mathit{k}\left(X\right)\mathit{k}{\left(X\right)}^T\right]{\tilde{\mathit{K}}}^{-1}\right\}.$$

If $c=0$, $D_{min}$ is the mean of the variance of the prediction by the associated Gaussian process [14].

Further upper-bounding the differential entropy $h\left(M_p\right)$ by the Gaussian entropy, we have the following theorem.

Theorem 5.

The rate distortion function $R_{\mathrm{fea}}\left(D\right)$ is upper-bounded as

$$R_{\mathrm{fea}}\left(D\right)\le R_{\mathrm{fea},G}\left(D\right)=\frac{1}{2}log\left|\mathit{I}+\frac{n-c\mathrm{tr}\left\{{\tilde{\mathit{K}}}^{-1}\right\}}{D-D_{min}\left(c\right)}{\tilde{\mathit{K}}}^{-1}\mathit{C}\right|,$$

(27)

where

$$\mathit{C}=E\left[\mathit{k}\left(X\right)\mathit{k}{\left(X\right)}^T\right]-E\left[\mathit{k}\left(X\right)\right]E{\left[\mathit{k}\left(X\right)\right]}^T.$$

(28)

The proof is put in Appendix B. In the simplest case where $\varphi \left(x\right)=x\in {\mathbb{R}}^1$, $n=1$, and the source is the Gaussian, $p\left(x\right)=N(x;0,{\sigma }^2)$, the upper bound in Equation (27) reduces to

$$R_{\mathrm{fea},G}\left(D\right)=\frac{1}{2}log\left(1+\frac{{\sigma }^2}{D}\right),$$

which is an asymptotically (as $D\to 0$) tight upper bound of the well-known rate distortion function for the Gaussian source under the squared distortion measure, $R\left(D\right)=\frac{1}{2}log\left(\frac{{\sigma }^2}{D}\right)$ [3,7].

5. Experimental Evaluation

We numerically evaluate the rate-distortion bounds obtained in the previous section. Designing a quantizer by the kernel $\mathtt{K}$-means algorithm, we compare its performance with the bounds.

We focus on the case of the Gaussian kernel,

$$K(x,y)=e^{{-\gamma \parallel x-y\parallel }^2}$$

(29)

with the kernel parameter $\gamma >0$.

5.1. Synthetic Data

As a source, we first assumed the uniform distribution on the union of the two regions, $C_1=\{x\in {\mathbb{R}}^m{;A\left(m\right)\parallel x\parallel }^m\le m/2\}$ and $C_2=\{x\in {\mathbb{R}}^m;m^2\le {A\left(m\right)\parallel x\parallel }^m\le m(m+1/2)\}$, where $C_1$ and $C_2$ have equal volumes and $C_1\cup C_2$ has volume 1. This suggests that $B={\left\{\frac{m(m+1/2)}{A\left(m\right)}\right\}}^{1/m}$ and $Q_B=1$ in Equation (10) and succeeding equations in Section 4.1 and Section 4.2.

We used the trapezoidal rule to compute the $one$-dimensional integrations in the lower bound $R_{\mathrm{inp},L}$ and the upper bound $R_{\mathrm{inp},G}$. We generated i.i.d sample of the size $n=200$ from the source to compute $\mathit{k}\left(x\right)$ and $\mathit{K}$ for $R_{\mathrm{fea},G}$ in Equation (27). Generating another 4000 data points, we approximated the required expectations. We optimized the regularization coefficient c to minimize the upper bound $R_{\mathrm{fea},G}$ for each D.

Using the same data set of the size 4000 as a training data set, we run the kernel $\mathtt{K}$-means algorithm 10 times with random initializations to obtain the minimum distortion for each rate. Varying the number $\mathtt{K}$ of quantized points from $2^1$ to $2^{10}$, for each $\mathtt{K}$, we counted the effective number ${\mathtt{K}}_{\mathtt{eff}}$ of quantized points which have at least one assigned data point and computed the rate by ${log}_2{\mathtt{K}}_{\mathtt{eff}}$ as the quantizer is first order, that is, the block length is one. The kernel parameter $\gamma$ was chosen so that the clear separation of $C_1$ and $C_2$ is obtained when $\mathtt{K}=2$.

After the training, we computed the distortion and rate for the test data set, by assigning each of 20,000 test data generated from the same source to the nearest quantized points in the feature space.

For each quantized point, we obtained its preimage. That is, if the kth quantized point is expressed as ${\sum }_{i=1}^n{\alpha }_{ki}\varphi \left(x_i\right)$, its preimage is

$$\begin{array}{ccc}\hfill y_k& =& \underset{y}{\mathrm{argmin}}{∥\varphi \left(y\right)-\sum _{i=1}^n{\alpha }_{ki}\varphi \left(x_i\right)∥}^2\hfill \\ \hfill & =& \underset{y}{\mathrm{argmax}}\sum _{i=1}^n{\alpha }_{ki}K(y,x_i).\hfill \end{array}$$

We used the mean shift procedure for the maximization, although this procedure only guarantees the convergence to a local maximum [15,16].

The obtained bounds and the quantizer performances are displayed in Figure 1a,b and for $m=2$ and $m=10$, respectively, in the forms of distortion-rate functions. The values of $D_{max}$ in Equations (7) and (9) are also indicated in the figures.

Figure 1. Rate-distortion bounds and quantizer performances for (a) $m=2$ and (b) $m=10$ [17].

In both dimensions, the upper bound $D_{\mathrm{fea},G}$ is smaller than $D_{\mathrm{inp},G}$ at low rates while the bound is above the quantizer performance. However, the value of $D_{max,\mathrm{fea}}$ suggests that the bound is informative at low rates. As the rate becomes higher, the lower and upper bounds of the input space reconstruction, $D_{L,\mathrm{inp}}$ and $D_{G,\mathrm{inp}}$, approach each other. In fact, they sandwich the quantizer performance tightly in the $two$-dimensional case, which suggests that the rate-distortion function for the feature space reconstruction, $R_{\mathrm{fea}}\left(D\right)$ is close to the rate-distortion function of the input space reconstruction $R_{\mathrm{inp}}\left(D\right)$ at high rates.

We see that the quantizer performances for $d_{\mathrm{fea}}$ and those for $d_{\mathrm{inp}}$ approach each other as the rate R grows. The upper bound $D_{\mathrm{inp},G}$ reasonably approximates the quantizer performance by the preimages, and it indicates that, in the $two$-dimensional case (Figure 1a), the results for $R=2$ and 3 bits can be improved by at least about 1 bit.

At low distortion levels, each source output should be reconstructed within a small neighborhood in the feature space where we can find another point y in the input space whose feature map $\varphi \left(y\right)$ is sufficiently close to the reconstruction. This suggests that the rate-distortion function of feature space reconstruction is well approximated by the rate-distortion function of input space reconstruction. In other words, combining multiple input points to make a reconstruction in feature space does not do any good for reducing distortion and only a single input point is enough when it is mapped into feature space. Hence, the rate-distortion bounds of input space reconstruction may be informative at low distortion levels.

In the 10-dimensional case (Figure 1b), the distortion in the test data set is close to $D_{\mathrm{inp},G}\left(R\right)$ or above it at high rates. This may be due to overfitting of the kernel $\mathtt{K}$-means to the training data set of the size, 4000. That is, as the the rate grows, the distortion in the training data set decreases and the discrepancy between the distortions in the training and test sets increases.

To examine the asymptotic behavior of $R_{\mathrm{inp}}\left(D\right)$ discussed in Section 4.3, we computed $R_{\mathrm{inp},L}\left(D\right)$ and $R_{\mathrm{inp},G}\left(D\right)$ for small D, that is, for large s. As well as the Gaussian kernel Equation (29), which has $\alpha =2$ in Equation (19), we applied the Laplacian kernel,

$$K(x,y)=e^{-\gamma \parallel x-y\parallel },$$

which corresponds to $\alpha =1$. The kernel parameter of the Laplacian kernel was set to the square root of the value used in the Gaussian kernel.

The rate-distortion bounds, $R_{\mathrm{inp},L}\left(D\right)$ and $R_{\mathrm{inp},G}\left(D\right)$ divided by $-(logD)/2$ for small distortion levels are shown in Figure 2a,b and for $m=2$ and $m=10$, respectively. We can see that, in each case, the ratio tends to $2m/\alpha$, that is, the rate-distortion dimension evaluated in Equation (22) as $D\to 0$. For the distortion levels smaller than those presented in Figure 2, the ratios start oscillating due to the errors of numerical integrations.

Figure 2. The ratios between the rate-distortion bounds and $-(logD)/2$ for (a) $m=2$ and (b) $m=10$. The bounds are for the Laplacian kernel ($\alpha =1$) and the Gaussian kernel ($\alpha =2$).

5.2. Image Data

We carried out a similar evaluation of the rate-distortion bounds and quantizer performances for a grayscale image data set extracted from the COIL20 data set [18]. We used the first category from 20 categories of images, which consisted of 72 images of size $32\times 32$. Dividing each $32\times 32$ image into small patches of size $2\times 2$ ($m=4$), we obtained 256 data from each image, and 18,432 data in total. Removing duplicate data points, we finally obtained 13,368 data. We used first 2048 data as the training data and the remaining 11,320 data as the test data. The training data set was also used for approximating expectations of kernel functions required to compute $R_{\mathrm{fea}}\left(D\right)$, and the first $n=256$ data points were used as the sample data in the definition of $d_{\mathrm{fea}}$. We evaluated only the upper bounds, $R_{\mathrm{fea},G}$ and $R_{\mathrm{inp},G}$, since the lower bound $R_{\mathrm{inp},L}$ requires estimating the source entropy from empirical data, which depends heavily on the estimation method, and hence is to be addressed more in detail.

Each dimension was normalized so that it has mean 0 and variance 1. Hence, $v_p$ in $R_{\mathrm{inp},G}$ was approximated by the empirical variance, 1. The boundary B in $R_{\mathrm{inp},G}$ was approximated by the maximum norm of the training data points.

The upper bounds and quantizer performances are presented in Figure 3. Although the upper bounds are loose and above the respective quantizer performances, the upper bound $D_{\mathrm{inp},G}\left(R\right)$ is roughly predictive of the quantizer performance in the input space, and so does $min\{D_{\mathrm{inp},G}\left(R\right),D_{\mathrm{fea},G}\left(R\right)\}$ for the reconstruction in the feature space.

Figure 3. Upper bounds of the rate-distortion functions and quantizer performance for image data.

6. Conclusions

In this paper, we have shown upper and lower bounds for the rate-distortion functions associated with kernel feature mapping. As suggested in Section 5, the upper bound for the reconstruction in feature space is informative at high distortion levels while the bounds for the reconstruction in input space are informative at low distortion levels. We have also evaluated the rate-distortion dimension of sources with bounded support under kernel-based distortion measures, which shows the asymptotic behavior of the rate-distortion function. Our future directions include deriving tighter bounds and exact evaluation of the rate-distortion function in some special cases. In particular, it is an important undertaking to derive a lower bound to the rate-distortion function of the reconstruction in feature space.