Next Article in Journal
Lean Path for High-Quality Development of Chinese Logistics Enterprises Based on Entropy and Gray Models
Previous Article in Journal
Energy and New Economic Approach for Nearly Zero Energy Hotels
Previous Article in Special Issue
Studying Lexical Dynamics and Language Change via Generalized Entropies: The Problem of Sample Size
Article Menu

Export Article

Open AccessArticle

Estimating Predictive Rate–Distortion Curves via Neural Variational Inference

1
Department of Linguistics, Stanford University, Stanford, CA 94305, USA
2
Department of Language Science, University of California, Irvine, CA 92697, USA
*
Author to whom correspondence should be addressed.
Entropy 2019, 21(7), 640; https://doi.org/10.3390/e21070640
Received: 13 May 2019 / Revised: 20 June 2019 / Accepted: 26 June 2019 / Published: 28 June 2019
(This article belongs to the Special Issue Information Theory and Language)
  |  
PDF [1391 KB, uploaded 2 July 2019]
  |     |  

Abstract

The Predictive Rate–Distortion curve quantifies the trade-off between compressing information about the past of a stochastic process and predicting its future accurately. Existing estimation methods for this curve work by clustering finite sequences of observations or by utilizing analytically known causal states. Neither type of approach scales to processes such as natural languages, which have large alphabets and long dependencies, and where the causal states are not known analytically. We describe Neural Predictive Rate–Distortion (NPRD), an estimation method that scales to such processes, leveraging the universal approximation capabilities of neural networks. Taking only time series data as input, the method computes a variational bound on the Predictive Rate–Distortion curve. We validate the method on processes where Predictive Rate–Distortion is analytically known. As an application, we provide bounds on the Predictive Rate–Distortion of natural language, improving on bounds provided by clustering sequences. Based on the results, we argue that the Predictive Rate–Distortion curve is more useful than the usual notion of statistical complexity for characterizing highly complex processes such as natural language. View Full-Text
Keywords: Predictive Rate–Distortion; natural language; information bottleneck; neural variational inference Predictive Rate–Distortion; natural language; information bottleneck; neural variational inference
Figures

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
SciFeed

Share & Cite This Article

MDPI and ACS Style

Hahn, M.; Futrell, R. Estimating Predictive Rate–Distortion Curves via Neural Variational Inference. Entropy 2019, 21, 640.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics

1

Comments

[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top