Next Article in Journal
Higher Order Geometric Theory of Information and Heat Based on Poly-Symplectic Geometry of Souriau Lie Groups Thermodynamics and Their Contextures: The Bedrock for Lie Group Machine Learning
Previous Article in Journal
Maximum Configuration Principle for Driven Systems with Arbitrary Driving
Open AccessArticle

Cross Entropy of Neural Language Models at Infinity—A New Bound of the Entropy Rate

1
Graduate School of Frontier Sciences, The University of Tokyo, Chiba 277-8561, Japan
2
Research Center for Advanced Science and Technology, The University of Tokyo, Tokyo 153-0041, Japan
*
Author to whom correspondence should be addressed.
Entropy 2018, 20(11), 839; https://doi.org/10.3390/e20110839
Received: 15 September 2018 / Revised: 23 October 2018 / Accepted: 23 October 2018 / Published: 2 November 2018
(This article belongs to the Section Complexity)
Neural language models have drawn a lot of attention for their strong ability to predict natural language text. In this paper, we estimate the entropy rate of natural language with state-of-the-art neural language models. To obtain the estimate, we consider the cross entropy, a measure of the prediction accuracy of neural language models, under the theoretically ideal conditions that they are trained with an infinitely large dataset and receive an infinitely long context for prediction. We empirically verify that the effects of the two parameters, the training data size and context length, on the cross entropy consistently obey a power-law decay with a positive constant for two different state-of-the-art neural language models with different language datasets. Based on the verification, we obtained 1.12 bits per character for English by extrapolating the two parameters to infinity. This result suggests that the upper bound of the entropy rate of natural language is potentially smaller than the previously reported values. View Full-Text
Keywords: entropy rate; natural language; language model; neural language model entropy rate; natural language; language model; neural language model
Show Figures

Graphical abstract

MDPI and ACS Style

Takahashi, S.; Tanaka-Ishii, K. Cross Entropy of Neural Language Models at Infinity—A New Bound of the Entropy Rate. Entropy 2018, 20, 839.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Search more from Scilit
 
Search
Back to TopTop