# Rate Distortion Functions and Rate Distortion Function Lower Bounds for Real-World Sources

## Abstract

**:**

## 1. Introduction

## 2. Source Models

## 3. Distortion Measures

## 4. Rate Distortion Theory

**Theorem**

**1**(Shannon’s third theorem)

**.**

**Theorem**

**2.**

#### 4.1. Memoryless Gaussian Sources

#### 4.2. Reverse Water-Filling

**Theorem**

**3**(Reverse water-filling theorem)

**.**

**Proof.**

#### 4.3. Rate Distortion Function for a Gaussian Autoregressive Source

**Theorem**

**4.**

#### 4.4. Rate Distortion Bounds for Composite Source Models

#### 4.5. The Shannon Lower Bound

#### 4.6. The Wyner–Ziv Lower Bound

## 5. Theoretical Rate-Distortion Performance of a Specific Codec Structure

## 6. Operational Rate Distortion Functions

## 7. Image and Video Models

**separable**model

**isotropic**model was proposed as

## 8. Rate Distortion Bounds for Video

## 9. Speech Models

## 10. Rate Distortion Functions for Speech

## 11. R(D) and Codec Design

#### 11.1. $R(D)$ for Real-World Sources

#### 11.2. Codec Design Approach

#### 11.3. Small Distortion Lower Bounds

## 12. Conclusions

## Acknowledgments

## Conflicts of Interest

## References

- Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J.
**1948**, 27, 379–423. [Google Scholar] [CrossRef] - Shannon, C.E. Coding Theorems for a Discrete Source with a Fidelity Criterion. IRE Conv. Rec.
**1959**, 7, 142–163. [Google Scholar] - Gallager, R.G. Information Theory and Reliable Communication; John Wiley & Sons, Inc.: New York, NY, USA, 1968. [Google Scholar]
- Netravali, A.N.; Haskell, B.G. Digital Pictures: Representation and Compression; Plenum Press: New York, NY, USA, 1988. [Google Scholar]
- Ortega, A.; Ramchandran, K. Rate-distortion methods for image and video compression. IEEE Signal Process. Mag.
**1998**, 15, 23–50. [Google Scholar] [CrossRef] - Effros, M. Optimal Modeling for Complex System Design: The Lessons of Rate Distortion Theory. IEEE Signal Process. Mag.
**1998**, 15, 51–73. [Google Scholar] [CrossRef] - Gray, R.M. Conditional Rate-Distortion Theory; Technical Report 6502-2; Stanford Electron. Lab.: Stanford, CA, USA, 1972. [Google Scholar]
- Gibson, J.D.; Hu, J. Rate distortion bounds for voice and video. Found. Trends Commun. Inf. Theory
**2014**, 10, 379–514. [Google Scholar] [CrossRef] - Sze, V.; Budagavi, M.; Sullivan, G.J. High efficiency video coding (HEVC). In Integrated Circuit and Systems, Algorithms and Architectures; Springer: New York, NY, USA, 2014; pp. 1–375. [Google Scholar]
- Berger, T. Rate Distortion Theory; Prentice-Hall: Upper Saddle River, NJ, USA, 1971. [Google Scholar]
- ITU-T Recommendation P.862. Perceptual Evaluation of Speech Quality (PESQ), An Objective Method for End-to-End Speech Quality Assessment of Narrow-Band Telephone Networks and Speech Codecs. 2001. Available online: https://www.itu.int/rec/T-REC-P.862 (accessed on 8 November 2017).
- ITU-T Recommendation P.862.2. Wideband Extension to Recommendation P.862 for the Assessment of Wideband Telephone Networks and Speech Codecs. 2007. Available online: https://www.itu.int/rec/T-REC-P.862.2-200511-S/en (accessed on 8 November 2017).
- Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley-Interscience: Hoboken, NJ, USA, 2006. [Google Scholar]
- Gray, R.M.; Davisson, L.D. A Mathematical Theory of Data Compression? In Proceedings of the International Conference on Communications, 1974; pp. 40A-1–40A-5. [Google Scholar]
- Gray, R.M. Source Coding Theory; Springer: New York, NY, USA, 1990. [Google Scholar]
- Girod, B. The efficiency of motion-compensating prediction for hybrid coding of video sequences. IEEE J. Sel. Areas Commun.
**1987**, 5, 1140–1154. [Google Scholar] [CrossRef] - Flierl, M.; Girod, B. Multiview Video Compression. IEEE Signal Process. Mag.
**2007**, 24, 66–76. [Google Scholar] [CrossRef] - Flierl, M.; Mavlankar, A.; Girod, B. Motion and Disparity Compensated Coding for Multiview Video. IEEE Trans. Circuits Syst. Video Technol.
**2007**, 17, 1474–1484. [Google Scholar] [CrossRef] - Kalveram, H.; Meissner, P. Rate Distortion Bounds for Speech Waveforms based on Itakura–Saito-Segmentation. Signal Process. IV Theor. Appl.
**EURASIP**.**1988**. [Google Scholar] - Kalveram, H.; Meissner, P. Itakura–Saito clustering and rate distortion functions for a composite source model of speech. Signal Process.
**1989**, 18, 195–216. [Google Scholar] [CrossRef] - Gray, R.M. Information rates of autoregressive processes. IEEE Trans. Inf. Theory
**1970**, 16, 412–421. [Google Scholar] [CrossRef] - Fontana, R. A Class of Composite Sources and Their Ergodic and Information Theoretic Properties. Ph.D. Thesis, Standford University, Standford, CA, USA, 1978. [Google Scholar]
- Garde, S. Communication of Composite Sources. Ph.D. Thesis, Univeristy of California, Berkeley, CA, USA, 1980. [Google Scholar]
- Naraghi-Pour, M.; Hegde, M.; Arora, N. DPCM encoding of regenerative composite processes. IEEE Trans. Inf. Theory
**1994**, 40, 153–160. [Google Scholar] [CrossRef] - Carter, M. Source Coding of Composite Sources. Ph.D. Thesis, The University of Michigan, Ann Arbor, MI, USA, 1984. [Google Scholar]
- Tziritas, G. Rate distortion theory for image and video coding. In Proceedings of the International Conference on Digital Signal Processing, Limassol, Cyprus, 26–28 June 1995. [Google Scholar]
- Mitrakos, D.; Constantinides, A. Maximum likelihood estimation of composite source models for image coding. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Boston, MA, USA, 14–16 April 1983; Volume 8, pp. 1244–1247. [Google Scholar]
- Gibson, J.D.; Hu, J.; Ramadas, P. New Rate Distortion Bounds for Speech Coding Based on Composite Source Models. In Proceedings of the Information Theory and Applications Workshop (ITA), San Diego, CA, USA, 31 January–5 February 2010. [Google Scholar]
- Gray, R.M. A new class of lower bounds to information rates of stationary sources via conditional rate-distortion functions. IEEE Trans. Inf. Theory
**1973**, 19, 480–489. [Google Scholar] [CrossRef] - Wyner, A.D.; Ziv, J. Bounds on the Rate-Distortion Function for Stationary Sources with Memory. IEEE Trans. Inf. Theory
**1971**, 17, 508–513. [Google Scholar] [CrossRef] - Grenander, U.; Rosenblatt, M. Statistical Analysis of Stationary Time Series; Wiley: New York, NY, USA, 1957. [Google Scholar]
- Hu, J.; Gibson, J.D. Rate distortion bounds for blocking and intra-frame prediction in videos. In Proceedings of the Information Theory and Applications Workshop, University of California, San Diego, CA, USA, 8–13 February 2009. [Google Scholar]
- Habibi, A.; Wintz, P.A. Image coding by linear transformation and block quantization. IEEE Trans. Commun. Technol.
**1971**, 19, 50–62. [Google Scholar] [CrossRef] - O’Neal, J.B., Jr.; Natarajan, T.R. Coding isotropic images. IEEE Trans. Inf. Theory
**1977**, 23, 697–707. [Google Scholar] [CrossRef] - Clarke, R.J. Transform Coding of Images; Academic Press: London, UK, 1985. [Google Scholar]
- Tasto, M.; Wintz, P.A. A bound on the rate-distortion function and application to images. IEEE Trans. Inf. Theory
**1972**, 18, 150–159. [Google Scholar] - Hu, J.; Gibson, J.D. New Rate Distortion Bounds for Natural Videos Based on a Texture-Dependent Correlation Model. IEEE Trans. Circuits Syst. Video Technol.
**2009**, 19, 1081–1094. [Google Scholar] - Hu, J.; Gibson, J.D. New rate distortion bounds for natural videos based on a texture dependent correlation model in the spatial-temporal domain. In Proceedings of the 46th Annual Allerton Conference on Communication, Controls, and Computing, Urbana-Champaign, IL, USA, 23–26 September 2008. [Google Scholar]
- Hu, J.; Gibson, J.D. New rate distortion bounds for natural videos based on a texture dependent correlation model. In Proceedings of the IEEE international Symposium on Information Theory, Nice, France, 24–29 June 2007. [Google Scholar]
- Hu, J.; Gibson, J.D. New Block-Based Local-Texture-Dependent Correlation Model of Digitized Natural Video. In Proceedings of the 40th Annual Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA, 29 October–1 November 2006. [Google Scholar]
- Bross, B.; Han, W.J.; Ohm, J.R.; Sullivan, G.J.; Wiegand, T. High Efficiency Video Coding (HEVC) text specification draft 9 (JCTVC-K1003 v10). 2012. [Google Scholar]
- Gibson, J.D.; Berger, T.; Lookabaugh, T.; Lindbergh, D.; Baker, R.L. Digital Compression for Multimedia: Principles and Standards; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 1998. [Google Scholar]
- Gibson, J.D. Speech Compression. Information
**2016**, 7, 32. [Google Scholar] [CrossRef] - Wang, S.; Gersho, A. Phonetically-based vector excitation coding of speech at 3.6 kbit/s. In Proceedings of the 1989 International Conference on Acoustics, Speech, and Signal Processing, ICASSP-89, Glasgow, UK, 23–26 May 1989; pp. 49–52. [Google Scholar]
- Wang, S.; Gersho, A. Improved Phonetically-Segmented Vector Excitation Coding at 3.4 Kb/s. In Proceedings of the 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP-92), San Francisco, CA, USA, 23–26 March 1992. [Google Scholar]
- Das, A.; Gersho, A. A variable-rate natural-quality parametric speech coder. In Proceedings of the IEEE International Conference on Communications, SUPERCOMM/ICC’94, Conference Record, Serving Humanity Through Communications, New Orleans, LA, USA, 1–5 May 1994; Volume 1, pp. 216–220. [Google Scholar]
- Das, A.; Paksoy, E.; Gersho, A. Multimode and Variable-Rate Coding of Speech. In Speech Coding and Synthesis; Kleijn, W., Paliwal, K., Eds.; Elsevier Science: New York, NY, USA, 1995; pp. 257–288. [Google Scholar]
- Gibson, J.D.; Li, Y.Y. Rate Distortion Performance Bounds for Wideband Speech. In Proceedings of the Information Theory and Applications Workshop (ITA), San Diego, CA, USA, 5–10 February 2012. [Google Scholar]
- Atal, B.S.; Hanauer, S.L. Speech analysis and synthesis by linear prediction of the speech wave. J. Acoust. Soc. Am.
**1971**, 50, 637–655. [Google Scholar] [CrossRef] [PubMed]

**Figure 1.**Marginal rate distortion functions for different local textures, ${R}_{V|Y=y}({D}_{y})$, for a frame in paris.cif (from [8]).

**Figure 2.**Comparison of rate distortion bounds and the operational rate distortion curves of AVC/H.246 and HEVC for inter-frame coding for the first 5 frames of paris.cif (from [8]).

**Figure 6.**The rate distortion bounds, operational rate distortion performance of speech codecs, and small distortion lower bound for the narrowband sequence “A lathe is a big tool.” (adapted from [8]).

**Figure 7.**The rate distortion bounds, operational rate distortion performance of speech codecs, and small distortion lower bound for the narrowband sequence “We were away a year ago.”(adapted from [8].

**Table 1.**Composite Source Models for Wideband Speech Sentences (from [48]).

Sequence | Mode | Autocorrelation Coefficients for V, ON, H Average Frame Energy for UV | Mean Square Prediction Error | Probability |
---|---|---|---|---|

F1 (Female) (active speech level: $-25.968$ dBov) (sampling rate: 12.8 kHz) | V | [1 $0.8448$ $0.5891$ $0.4132$ $0.3156$ $0.2670$ $0.2122$ $0.1462$ $0.0599$ $-0.0987$ $-0.3028$ $-0.4109$ $-0.3816$ $-0.3084$ $-0.2673$ $-0.2879$ $-0.3293$] | $0.0253$ | $0.4406$ |

ON | [1 $0.1226$ $-0.2917$ $0.2239$ $-0.0034$] | $0.5241$ | $0.0043$ | |

H | 0 | |||

UV | $0.0009$ | $0.0009$ | $0.0028$ | |

S | $0.5523$ | |||

M3 (Male) (active speech level: $-29.654$ dBov) (sampling rate: 12.8 kHz) | V | [1 $0.7954$ $0.6612$ $0.4775$ $0.2864$ $0.2398$ $0.2004$ $0.2169$ $0.2214$ $0.2248$ $0.2022$ $0.1613$ $0.1333$ $0.1075$ $0.1334$ $0.1759$ $0.1662$] | $0.0861$ | $0.6939$ |

ON | [1 $0.9564$ $0.9334$ $0.9104$ $0.8862$] | $0.0066$ | $0.0069$ | |

H | [1 $0.9387$ $0.9028$ $0.8696$ $0.8257$] | $0.0129$ | $0.0461$ | |

UV | $0.0015$ | $0.0015$ | $0.0064$ | |

S | $0.2467$ |

© 2017 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Gibson, J.
Rate Distortion Functions and Rate Distortion Function Lower Bounds for Real-World Sources. *Entropy* **2017**, *19*, 604.
https://doi.org/10.3390/e19110604

**AMA Style**

Gibson J.
Rate Distortion Functions and Rate Distortion Function Lower Bounds for Real-World Sources. *Entropy*. 2017; 19(11):604.
https://doi.org/10.3390/e19110604

**Chicago/Turabian Style**

Gibson, Jerry.
2017. "Rate Distortion Functions and Rate Distortion Function Lower Bounds for Real-World Sources" *Entropy* 19, no. 11: 604.
https://doi.org/10.3390/e19110604