Next Article in Journal
Boltzmann Sampling by Degenerate Optical Parametric Oscillator Network for Structure-Based Virtual Screening
Previous Article in Journal
The Shell Collapsar—A Possible Alternative to Black Holes
Article

Entropy Rate Estimates for Natural Language—A New Extrapolation of Compressed Large-Scale Corpora

1
Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan
2
Research Center for Advanced Science and Technology, University of Tokyo, Tokyo 153-8904, Japan
3
Institute of Computer Science, Polish Academy of Sciences, Warszawa 01-248, Poland
*
Author to whom correspondence should be addressed.
Academic Editor: J. A. Tenreiro Machado
Entropy 2016, 18(10), 364; https://doi.org/10.3390/e18100364
Received: 9 September 2016 / Revised: 30 September 2016 / Accepted: 9 October 2016 / Published: 12 October 2016
(This article belongs to the Section Complexity)
One of the fundamental questions about human language is whether its entropy rate is positive. The entropy rate measures the average amount of information communicated per unit time. The question about the entropy of language dates back to experiments by Shannon in 1951, but in 1990 Hilberg raised doubt regarding a correct interpretation of these experiments. This article provides an in-depth empirical analysis, using 20 corpora of up to 7.8 gigabytes across six languages (English, French, Russian, Korean, Chinese, and Japanese), to conclude that the entropy rate is positive. To obtain the estimates for data length tending to infinity, we use an extrapolation function given by an ansatz. Whereas some ansatzes were proposed previously, here we use a new stretched exponential extrapolation function that has a smaller error of fit. Thus, we conclude that the entropy rates of human languages are positive but approximately 20% smaller than without extrapolation. Although the entropy rate estimates depend on the script kind, the exponent of the ansatz function turns out to be constant across different languages and governs the complexity of natural language in general. In other words, in spite of typological differences, all languages seem equally hard to learn, which partly confirms Hilberg’s hypothesis. View Full-Text
Keywords: entropy rate; universal compression; stretched exponential; language universals entropy rate; universal compression; stretched exponential; language universals
Show Figures

Figure 1

MDPI and ACS Style

Takahira, R.; Tanaka-Ishii, K.; Dębowski, Ł. Entropy Rate Estimates for Natural Language—A New Extrapolation of Compressed Large-Scale Corpora. Entropy 2016, 18, 364. https://doi.org/10.3390/e18100364

AMA Style

Takahira R, Tanaka-Ishii K, Dębowski Ł. Entropy Rate Estimates for Natural Language—A New Extrapolation of Compressed Large-Scale Corpora. Entropy. 2016; 18(10):364. https://doi.org/10.3390/e18100364

Chicago/Turabian Style

Takahira, Ryosuke, Kumiko Tanaka-Ishii, and Łukasz Dębowski. 2016. "Entropy Rate Estimates for Natural Language—A New Extrapolation of Compressed Large-Scale Corpora" Entropy 18, no. 10: 364. https://doi.org/10.3390/e18100364

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop