Open Access This article is
- freely available
Entropy 2018, 20(8), 608; doi:10.3390/e20080608
New Estimations for Shannon and Zipf–Mandelbrot Entropies
College of Science, Hunan City University, Yiyang 413000, China
Department of Mathematics, University of Peshawar, Peshawar 25000, Pakistan
Department of Mathematics, University of Sa’adah, Sa’adah 1872, Yemen
Department of Mathematics, Huzhou University, Huzhou 313000, China
Author to whom correspondence should be addressed.
Received: 9 July 2018 / Accepted: 14 August 2018 / Published: 16 August 2018
The main purpose of this paper is to find new estimations for the Shannon and Zipf–Mandelbrot entropies. We apply some refinements of the Jensen inequality to obtain different bounds for these entropies. Initially, we use a precise convex function in the refinement of the Jensen inequality and then tamper the weight and domain of the function to obtain general bounds for the Shannon entropy (SE). As particular cases of these general bounds, we derive some bounds for the Shannon entropy (SE) which are, in fact, the applications of some other well-known refinements of the Jensen inequality. Finally, we derive different estimations for the Zipf–Mandelbrot entropy (ZME) by using the new bounds of the Shannon entropy for the Zipf–Mandelbrot law (ZML). We also discuss particular cases and the bounds related to two different parametrics of the Zipf–Mandelbrot entropy. At the end of the paper we give some applications in linguistics.
Keywords:Jensen inequality; convex function; Shannon entropy; Zipf–Mandelbrot entropy
The idea of the Shannon entropy  plays a key role in information theory, while in some cases, it is denoted as measure of uncertainty. There are basically two methods for understanding the Shannon entropy. Under one point of view, the Shannon entropy quantifies the amount of information in regard to the value of X (after measurement). Under another point of view, the Shannon entropy tells us the amount of uncertainty about the variable of X before we learn its value (before measurement) . The random variable, entropy, is characterized regarding its probability distribution and it can appear as a better measure of predictability or uncertainty. SE permits the appraisal of the normal least number of bits expected to encode a series of symbols based on the letters in order of estimation and the recurrence of the symbols. The formula for SE is given by where with .
There are many applications of the Shannon entropy in most applied sciences and in other sciences, such as biology , genomic geography , and finance . Currently, the Shannon entropy is applied in the simulation of laser dynamics and as an objective measure to evaluate models and compare observational results [6,7].
In 1932, George Zipf gave the idea that the size of the th largest occurrence of an event is inversely proportional to its rank. That is, this law states that , where is the frequency of occurrence of the th ranked and b is close to unity. As in linguistics, Zipf found that one can calculate the number of times each word appears in the text. Therefore, if the rank () of the word is in accordance with the frequency of the word’s appearance (), then the product of these two numbers is a constant (: , see [8,9]).
There are several applications of the Zipf law, and here we present some of them. This law has been used in city populations. Kristian Giesen and Jens Suedekum conducted a study on the city measure distributions of single German districts’ reliance on researching the phenomenon . They built their study based on the intuition by Gabaix (1999) which states that the Zipf law takes after an irregular development process. This means that Gabaix displays that if the districts follow the Gibrat law, they should notice the Zipf law at both the districts at a national level. By utilizing non-parametric procudures, they found that the Gibrat law holds in each German district, regardless of how “districts” are defined. To put it differently, the Gibrat and Zipf laws are inclined to hold ubiquitously in space. In geology, the Zipf law has been used with temperate prosperity in the resource estimation of extracting ores from the ground and petroleum . In principle, it forecasts how many entities of a confident size can be left in a sequence of decreasing size, assuming the largest has been established. The solar flare intensity (M. E. J. Newman, 2004)  represents the cumulative distribution of the vertex gamma-ray density of solar flares, for which perceptions were made between 1980 and 1989 by the well-known HardX-Ray fulmination spectrometer onboard the solar maximum mission satellite launched in 1980. The spectrometer uses a CsI gleaming discloser to measure gamma-rays from solar flares. For website traffic (Shane Parkins, 2015) , the Zipf law seems, by all accounts, to be the control as opposed to the special case. It is available at the level of routers that transmit data from one geographic location to another and in the content of the World Wide Web. At the social and economic levels, it also determines how persons choose the sites they visit and form peer-to-peer societies. The omnipresent nature of the Zipf law in cyberspace is geared toward deeper empathy with the internet phenomena, for example, discovering the potential of prevalence proxy caches in divergent Autonomous Systems (ASes) with the purpose of reducing the costs incurred by internet service providers and pacification of the load on the internet backbone .
It was determined that Zipf’s law can describe the size and rank distribution of earthquakes, including those with magnitude, but it cannot predict when they will occur. In the earth-moon the crater size-frequency distribution can be represented by the Zipf law [12,15].
In 1966, Benoit Mandelbrot gave an enhancement for the Zipf law, known as ZML, which gives a generalization of the account of the low-rank words in corpus :where and if , we get the Zipf law.
If , , and , then the Zipf–Mandelbrot law (probability mass function) is defined by
The formula for ZME is given bywhere .
There are many applications of ZML which can be found in linguistics [16,17], ecological field studies , and information sciences . Recently, the Zipf–Mandelbrot law was applied to various types of f-divergences and distances, for example Kullback–Leibler divergence, Bhattacharyya distance (via coefficient), Hellinger distance, -divergence, etc .
To complete this section, we give some notions and results from ref. .
Let be a convex function defined on the convex set , . Let s be fixed positive integer and l be all those positive integers, such that . Suppose represents any subsets of , such that and for , where , , with , and for any , , we represent and for positive real numbers . For the convex function g and the n-tuples , , the following functional is defined:
Particularly, for , we have
Analogously, for other particular values of s with , one can obtain different functionals.
(). Let be a convex function defined on the convex set . If and for with , then we have
(). Let be a convex function defined on the convex set , , and for with . Then,
For some other results related to the Jensen inequality and the Shannon and Zipf–Mandelbrot entropies, see refs. [8,22,23,24].
Due to the great importance of the Shannon and Zipf–Mandelbrot entropies, many results are devoted to these entropies in the literature. The main focus of this paper was to associate some refinements of the Jensen inequality to the Shannon and Zipf–Mandelbrot entropies. In this paper, we use the main results given in ref.  and obtain some estimations for these entropies. We also discuss some particular cases of these results. At the end of the paper, we give some applications in linguistics. The idea of this paper can be applied for other results of the Jensen inequality to obtain new estimations for these entropies.
2. Estimations for the Shannon Entropy
We start by giving our first main result for the Shannon entropy.
Let , where , with . Then, the following inequalities hold:
Let , where , with , then
Let , with , then
In the following corollary, we discuss another particular case of Theorem 3.
Let for with , then
It is obvious that
3. Estimations for the Zipf–Mandelbrot Entropy
In the following main result, we obtain some general estimations for the Zipf–Mandelbrot entropy.
Let , , , , , then
If we substitute with , , we have
Then,where and .
Now, by applying Theorem 3 for , we obtain the required result. ☐
Let , , , for , then
Let , , , then
Let , , , then
Using in (12), we obtain Corollary 6. ☐
By using Remark 2, we also have
In the following result, we obtain the estimation for the Zipf–Mandelbrot entropy concerning two different parameters.
Let , , then
Let , , . Then, using the proof of Theorem 4, we get
Therefore, using (6) for and , , we obtain (13). ☐
Let , , , then
By taking in (13), we obtain (14). ☐
Let , , then
Using in (14), we obtain (15). ☐
Now we give applications of the above results in linguistics.
In ref. , Gelbukh and Sidorov observed the difference between the coefficients and in the Zipf law for the English and Russian languages. They processed 39 literature texts for each language, chosen randomly from different genres, with the requirement that the size be greater than 10,000 running words each. They calculated the coefficients for each of the mentioned texts and as a result, they obtained an average of for the English language and for the Russian language.
In the following results, we give the application of inequality (11) for the English language.
Let , for Then, we have
Similarly, we can give an application for the Russian language.
Now we give an application for the result of the related two parameters: for the English language and for the Russian language, which is in fact application of the inequality (14).
Let . Then, we have
In a similar way, applications of the remaining results from Section 3 can be given.
All authors contributed equally to the final manuscript.
The research was supported by the Natural Science Foundation of China (Grants No. 61673169, No. 11601485, No. 11701176) and the Natural Science Foundation of the Department of Education of Zhejiang Province (Grant No. Y201635325).
The authors express their sincere thanks to the referees for careful reading of the manuscript and very helpful suggestions that improved the current manuscript substantially.
Conflicts of Interest
The authors declare no conflict of interest.
- Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 623–656. [Google Scholar] [CrossRef]
- Latif, N.; Pečarić, Đ.; Pečarić, J. Majorization, Csiszár divergence and Zipf-Mandelbrot law. J. Inequal. Appl. 2017, 2017, 197. [Google Scholar] [CrossRef] [PubMed]
- Quastler, H. (Ed.) Essays on the Use of Information Theory in Biology; University of Illinois: Urbana, IL, USA, 1953. [Google Scholar]
- Sherwin, W.B. Entropy and information approaches to genetic diversity and its expression: Genomic geography. Entropy 2010, 12, 1765–1798. [Google Scholar] [CrossRef]
- Zhou, R.-X.; Cai, R.; Tong, G.-Q. Applications of entropy in finance: A review. Entropy 2013, 15, 4909–4931. [Google Scholar] [CrossRef]
- Guisado, J.L.; Jiménez-Morales, F.; Guerra, J.M. Application of shannon’s entropy to classify emergent behaviors in a simulation of laser dynamics. Math. Comput. Model. 2005, 42, 847–854. [Google Scholar] [CrossRef]
- Wellmann, J.F.; Regenauer-Lieb, K. Uncertainties have a meaning: Information entropy as a quality measure for 3-D geological models. Tectonophysics 2012, 526–529, 207–216. [Google Scholar] [CrossRef]
- Adil Khan, M.; Pečarić, Đ.; Pečarić, J. Bounds for Shannon and Zipf-Mandelbrot entropies. Math. Methods Appl. Sci. 2017, 40, 7316–7322. [Google Scholar] [CrossRef]
- Silagadze, Z.K. Citations and the Zipf-Mandelbrot’s law. Complex Syst. 1997, 11, 487–499. [Google Scholar]
- Kristian, G.; Jens, S. Zipf’s law for cities in the regions and the country. J. Econ. Geogr. 2010, 11, 667–686. [Google Scholar]
- Merriam, D.F.; Drew, L.J.; Schuenemeyer, J.H. Zipf’s law: A viable geological paradigm? Nat. Resour. Res. 2004, 13, 265–271. [Google Scholar] [CrossRef]
- Newman, M.E.J. Power laws, Pareto distributions and Zipf’s law. Contemp. Phys. 2005, 46, 323–351. [Google Scholar] [CrossRef]
- Parkins, S. Website Traffic and Zipf’s Law. 2015. Available online: http://www.linkedin.com/pulse/website-traffic-zipfs-law-shane-parkins?articleId=6064338455452807169 (accessed on 26 October 2015).
- Hefeeda, M.; Saleh, O. Traffic modeling and proportional partial caching for peer-to-peer systems. IEEE/ACM Trans. Netw. 2008, 16, 1447–1460. [Google Scholar] [CrossRef]
- Neukum, G.; Ivanov, B.A. Crater size distributions and impact probabilities on earth from lunar, terrestrial-planet, and asteroid cratering data. In Hazards Due to Comets & Asteroids; University Arizona Press: Tucson, AZ, USA, 1994. [Google Scholar]
- Montemurro, M.A. Beyond the Zipf-Mandelbrot law in quantitative linguistics. Phys. A Stat. Mech. Its Appl. 2001, 300, 567–578. [Google Scholar] [CrossRef]
- Manin, D.Y. Mandelbrot’s model for Zipf’s law: Can mandelbrot’s model explain Zipf’s law for language? J. Quant. Ling. 2009, 16, 274–285. [Google Scholar] [CrossRef]
- Mouillot, D.; Lepretre, A. Introduction of relative abundance distribution (RAD) indices, estimated from the rank-frequency diagrams (RFD), to assess changes in community diversity. Environ. Monit. Assess. 2000, 63, 279–295. [Google Scholar] [CrossRef]
- Lovričević, N.; Pečarić, Đ.; Pečarić, J. Zipf-Mandelbrot law, f-divergences and the Jensen-type interpolating inequalities. J. Inequal. Appl. 2018, 2018, 36. [Google Scholar]
- Adil Khan, M.; Ali Khan, G.; Alia, T.; Kilicman, A. On the refinement of Jensen’s inequality. Appl. Math. Comput. 2015, 262, 128–135. [Google Scholar]
- Dragomir, S.S. A refinement of Jensen’s inequality with applications for f-divergence measures. Taiwan. J. Math. 2010, 14, 153–164. [Google Scholar] [CrossRef]
- Adil Khan, M.; Pečaric, Đ.; Pečarić, J. On Zipf-Mandelbrot entropy. J. Comput. Appl. Math. 2019, 346, 192–204. [Google Scholar] [CrossRef]
- Dragomir, S.S. Bounds for the normalised Jensen functional. Bull. Aust. Math. Soc. 2006, 74, 471–478. [Google Scholar] [CrossRef]
- Abbaszadeh, S.; Gordji, M.E.; Pap, E.; Szakal, A. Jensen-type inequalities for Sugeno integral. Inf. Sci. 2017, 376, 148–157. [Google Scholar] [CrossRef]
- Dragomir, S.S. A new refinement of Jensen’s inequality in linear spaces with application. Math. Comput. Model. 2010, 52, 1497–1505. [Google Scholar] [CrossRef]
- Gelbukh, A.; Sidorov, G. Zipf and Heaps laws’ coefficients depend on language. Lect. Notes Comput. Sci. 2001, 2004, 332–335. [Google Scholar]
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).