# Kullback-Leibler Divergence and Mutual Information of Experiments in the Fuzzy Case

## Abstract

**:**

## 1. Introduction

## 2. Basic Definitions and Facts

**Definition 1**

**.**By a fuzzy partition (of a space $(\Omega ,M,\mu )$) we understand a finite collection $\xi =\left\{{f}_{1},\dots ,{f}_{n}\right\}$ of pairwise W-separated fuzzy subsets from M such that $\mu ({\cup}_{i=1}^{n}{f}_{i})=1$.

**Definition 2**

**.**Let $\xi =\left\{{f}_{1},\dots ,{f}_{n}\right\}$ and $\eta =\left\{{g}_{1},\dots ,{g}_{m}\right\}$ be two fuzzy partitions of a fuzzy probability space $(\Omega ,M,\mu ).$ A conditional entropy of $\eta $ given a fuzzy event ${f}_{i}\in \xi $ is defined by:

**Example**

**1.**

- (2.2)
- $\xi \prec \eta $ implies ${H}_{\mu}(\xi )\le {H}_{\mu}(\eta );$
- (2.3)
- ${H}_{\mu}(\eta \vee \varsigma /\xi )={H}_{\mu}(\varsigma /\xi \vee \eta )+{H}_{\mu}(\eta /\xi );$
- (2.4)
- $\xi \prec \eta $ implies ${H}_{\mu}(\xi /\varsigma )\le {H}_{\mu}(\eta /\varsigma );$
- (2.5)
- $\xi \prec \eta $ implies ${H}_{\mu}(\varsigma /\xi )\ge {H}_{\mu}(\varsigma /\eta );$
- (2.6)
- ${H}_{\mu}(\xi /\eta )\le {H}_{\mu}(\xi )$ with the equality if and only if $\xi ,\eta $ are statistically independent;
- (2.7)
- ${H}_{\mu}(\eta \vee \varsigma /\xi )\le {H}_{\mu}(\eta /\xi )+{H}_{\mu}(\varsigma /\xi );$
- (2.8)
- ${H}_{\mu}(\xi \vee \eta )={H}_{\mu}(\xi )+{H}_{\mu}(\eta /\xi ).$

**Definition 3**

**.**Let $\xi ,\eta $ be two fuzzy partitions of a given fuzzy probability space $(\Omega ,M,\mu ).$ The mutual information of $\xi $ and $\eta $ is defined by the formula:

**Theorem 1**

**.**Let $\xi =\left\{{f}_{1},\dots ,{f}_{n}\right\}$ and $\eta =\left\{{g}_{1},\dots ,{g}_{m}\right\}$ be two fuzzy partitions of a fuzzy probability space $(\Omega ,M,\mu ).$ Then:

**Theorem 2**

**.**${I}_{\mu}(\xi ,\eta )\ge 0$ with the equality if and only if $\xi ,\eta $ are statistically independent.

## 3. Mutual Information and Conditional Mutual Information in Fuzzy Probability Spaces

**Definition**

**4.**

**Remark**

**1.**

**Theorem**

**3.**

**Proof.**

**Theorem 4**(Chain rules)

**.**

- (i)
- ${H}_{\mu}({\xi}_{1}\vee {\xi}_{2}\vee \dots \vee {\xi}_{n})=$${\sum}_{i=1}^{n}{H}_{\mu}}({\xi}_{i}/{\vee}_{k=0}^{i-1}{\xi}_{k});$
- (ii)
- ${H}_{\mu}({\vee}_{i=1}^{n}{\xi}_{i}/\eta )$$={\displaystyle {\sum}_{i=1}^{n}{H}_{\mu}}({\xi}_{i}/({\vee}_{k=0}^{i-1}{\xi}_{k})\vee \eta );$
- (iii)
- ${I}_{\mu}({\vee}_{i=1}^{n}{\xi}_{i},\eta )=$${\sum}_{i=1}^{n}{I}_{\mu}}({\xi}_{i},\eta /{\vee}_{k=0}^{i-1}{\xi}_{k}).$

**Proof.**

**Definition**

**5.**

**Theorem**

**5.**

**Proof.**

**Theorem**

**6.**

- (i)
- ${I}_{\mu}(\xi \vee \eta ,\varsigma )=$${I}_{\mu}(\eta ,\varsigma );$
- (ii)
- ${I}_{\mu}(\eta ,\varsigma )=$${I}_{\mu}(\xi ,\varsigma )+$${I}_{\mu}(\varsigma ,\eta /\xi );$
- (iii)
- ${I}_{\mu}(\xi ,\eta /\varsigma )\le $${I}_{\mu}(\xi ,\eta );$
- (iv)
- ${I}_{\mu}(\xi ,\eta )\ge {I}_{\mu}(\xi ,\varsigma ).$

**Proof.**

- (i)
- Since by the assumption ${I}_{\mu}(\xi ,\varsigma /\eta )=0,$ using the chain rule for logical mutual information, we obtain:$${I}_{\mu}(\xi \vee \eta ,\varsigma )={I}_{\mu}(\eta \vee \xi ,\varsigma )={I}_{\mu}(\eta ,\varsigma )+{I}_{\mu}(\xi ,\varsigma /\eta )={I}_{\mu}(\eta ,\varsigma ).$$
- (ii)
- According to Theorem 3 we have ${I}_{\mu}(\xi \vee \eta ,\varsigma )=$${I}_{\mu}(\varsigma ,\xi )+$${I}_{\mu}(\varsigma ,\eta /\xi )$. Hence, using the equality (i) of this theorem, we obtain:$${I}_{\mu}(\eta ,\varsigma )={I}_{\mu}(\xi \vee \eta ,\varsigma )={I}_{\mu}(\varsigma ,\xi )+{I}_{\mu}(\varsigma ,\eta /\xi ).$$
- (iii)
- Since ${I}_{\mu}(\xi ,\eta )\ge 0,$ from (ii) it follows the inequality:$${I}_{\mu}(\varsigma ,\eta /\xi )\le {I}_{\mu}(\varsigma ,\eta ).$$By Theorem 5 we can interchange $\xi $ and $\varsigma $. Doing so we obtain ${I}_{\mu}(\xi ,\eta /\varsigma )\le $${I}_{\mu}(\xi ,\eta )$.
- (iv)
- By Theorem 3, the mutual information ${I}_{\mu}(\xi ,\eta \vee \varsigma )$ can be expressed in two different ways:$$\begin{array}{c}{I}_{\mu}(\xi ,\eta \vee \varsigma )={I}_{\mu}(\xi ,\eta )+{I}_{\mu}(\xi ,\varsigma /\eta )\\ ={I}_{\mu}(\xi ,\varsigma )+{I}_{\mu}(\xi ,\eta /\varsigma ).\end{array}$$Since $\xi \to \eta \to \varsigma $, we have ${I}_{\mu}(\xi ,\varsigma /\eta )=0$, and, therefore, it holds ${I}_{\mu}(\xi ,\eta )={I}_{\mu}(\xi ,\eta \vee \varsigma )$. Using the second equality, we obtain:$${I}_{\mu}(\xi ,\eta )={I}_{\mu}(\xi ,\eta \vee \varsigma )={I}_{\mu}(\xi ,\varsigma )+{I}_{\mu}(\xi ,\eta /\varsigma ).$$Since ${I}_{\mu}(\xi ,\eta /\varsigma )\ge 0$, we have ${I}_{\mu}(\xi ,\eta )\ge {I}_{\mu}(\xi ,\varsigma )$. □

## 4. Kullback-Leibler Divergence with Respect to Fuzzy P-Measures

**Definition**

**6.**

**Remark**

**2.**

**Example**

**2.**

**Theorem**

**7.**

**Proof.**

**Theorem**

**8.**

**Proof.**

**Corollary**

**1.**

**Proof.**

**Remark**

**3.**

- (i)
- A function $f$ is concave over an interval if and only if the function $-f$ is convex over the interval.
- (ii)
- The sum of two concave functions is itself concave; the sum of two convex functions is itself convex.
- (iii)
- Every real-valued affine function, i.e., each function of the form $f(x)=ax+b,$ $a,b\in \Re ,$ is simultaneously convex and concave.

**Proposition**

**1.**

**Theorem 9**(Concavity of entropy)

**.**

**Proof.**

**Theorem 10**(Convexity of K-L divergence)

**.**

**Proof.**

**Theorem**

**11.**

**Proof.**

**Definition**

**7.**

**Theorem 12**(Chain rule for K-L divergence)

**.**

**Proof.**

## 5. Discussion

## Acknowledgments

## Conflicts of Interest

## References

- Kolmogorov, A.N. Foundations of the Theory of Probability; Chelsea Press: New York, NY, USA, 1950. [Google Scholar]
- Shannon, C.E. A Mathematical Theory of Communication. Bell Syst. Tech. J.
**1948**, 27, 379–423. [Google Scholar] [CrossRef] - Markechová, D. The entropy of fuzzy dynamical systems and generators. Fuzzy Sets Syst.
**1992**, 48, 351–363. [Google Scholar] [CrossRef] - Piasecki, K. Probability of fuzzy events defined as denumerable additive measure. Fuzzy Sets Syst.
**1985**, 17, 271–284. [Google Scholar] [CrossRef] - Zadeh, L.A. Fuzzy Sets. Inf. Control
**1965**, 8, 338–358. [Google Scholar] [CrossRef] - Markechová, D. Entropy and mutual information of experiments in the fuzzy case. Neural Netw. World
**2013**, 23, 339–349. [Google Scholar] [CrossRef] - Markechová, D. Entropy of complete fuzzy partitions. Math. Slovaca
**1993**, 43, 1–10. [Google Scholar] - Markechová, D. A note to the Kolmogorov-Sinai entropy of fuzzy dynamical systems. Fuzzy Sets Syst.
**1994**, 64, 87–90. [Google Scholar] [CrossRef] - Ellerman, D. An Introduction to Logical Entropy and Its Relation to Shannon Entropy. Int. J. Seman. Comput.
**2013**, 7, 121–145. [Google Scholar] [CrossRef] - Markechová, D.; Riečan, B. Logical Entropy of Fuzzy Dynamical Systems. Entropy
**2016**, 18, 157. [Google Scholar] [CrossRef] - Ebrahimzadeh, A.; Giski, Z.E.; Markechová, D. Logical Entropy of Dynamical Systems—A General Model. Mathematics
**2017**, 5, 4. [Google Scholar] [CrossRef] - Mesiar, R.; Rybárik, J. Entropy of Fuzzy Partitions—A General Model. Fuzzy Sets Syst.
**1998**, 99, 73–79. [Google Scholar] [CrossRef] - Mesiar, R. The Bayes formula and the entropy of fuzzy probability spaces. Int. J. Gen. Syst.
**1990**, 4, 67–71. [Google Scholar] - Dumitrescu, D. Fuzzy measures and the entropy of fuzzy partitions. J. Math. Anal. Appl.
**1993**, 176, 359–373. [Google Scholar] [CrossRef] - Dumitrescu, D. Entropy of a fuzzy dynamical system. Fuzzy Sets Syst.
**1995**, 70, 45–57. [Google Scholar] [CrossRef] - Riečan, B. An entropy construction inspired by fuzzy sets. Soft Comput.
**2003**, 7, 486–488. [Google Scholar] - Hu, Q.; Yu, D.; Xie, Z.; Liu, J. Fuzzy probabilistic approximation spaces and their information measures. IEEE Trans. Fuzzy Syst.
**2006**, 14, 191–201. [Google Scholar] - Yu, D.; Hu, Q.; Wu, C. Uncertainty measures for fuzzy relations and their applications. Appl. Soft Comput.
**2007**, 7, 1135–1143. [Google Scholar] [CrossRef] - Rahimi, M.; Riazi, A. On local entropy of fuzzy partitions. Fuzzy Sets Syst.
**2014**, 234, 97–108. [Google Scholar] [CrossRef] - Rahimi, M.; Assari, A.; Ramezani, F. A Local Approach to Yager Entropy of Dynamical Systems. Int. J. Fuzzy Syst.
**2015**, 1, 1–10. [Google Scholar] [CrossRef] - Markechová, D.; Riečan, B. Entropy of Fuzzy Partitions and Entropy of Fuzzy Dynamical Systems. Entropy
**2016**, 18, 19. [Google Scholar] [CrossRef] - Khare, M. Fuzzy σ-algebras and conditional entropy. Fuzzy Sets Syst.
**1999**, 102, 287–292. [Google Scholar] [CrossRef] - Khare, M.; Roy, S. Conditional entropy and the Rokhlin metric on an orthomodular lattice with Bayessian state. Int. J. Theor. Phys.
**2008**, 47, 1386–1396. [Google Scholar] [CrossRef] - Srivastava, P.; Khare, M.; Srivastava, Y.K. M-Equivalence, entropy and F-dynamical systems. Fuzzy Sets Syst.
**2001**, 121, 275–283. [Google Scholar] [CrossRef] - Criado, F.; Gachechiladze, T. Entropy of fuzzy events. Fuzzy Sets Syst.
**1997**, 88, 99–106. [Google Scholar] [CrossRef] - Dubois, D.; Prade, M. A review of fuzzy set aggregation connectives. Inf. Sci.
**1985**, 36, 85–121. [Google Scholar] [CrossRef] - Gray, R.M. Entropy and Information Theory; Springer: Berlin/Heidelberg, Germany, 2009. [Google Scholar]
- Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat.
**1951**, 22, 79–86. [Google Scholar] [CrossRef] - Kullback, S. Information Theory and Statistics; John Wiley & Sons: New York, NY, USA, 1959. [Google Scholar]
- Schnakenberg, J. Network theory of microscopic and macroscopic behavior of master equation systems. Rev. Mod. Phys.
**1976**, 48, 571–585. [Google Scholar] [CrossRef] - Risken, H. The Fokker-Planck Equation, Methods of Solution and Applications; Springer: New York, NY, USA, 1984. [Google Scholar]
- Qian, H. Relative entropy: Free energy associated with equilibrium fluctuations and nonequilibrium deviations. Phys. Rev. E
**2001**, 63, 042103. [Google Scholar] [CrossRef] [PubMed] - Ellis, R.S. Entropy, Large Deviations, and Statistical Mechanics; Springer: New York, NY, USA, 1985. [Google Scholar]
- Dvurečenskij, A. On the existence of probability measures on fuzzy measurable spaces. Fuzzy Sets Syst.
**1991**, 43, 173–181. [Google Scholar] [CrossRef] - Piasecki, K. New concept of separated fuzzy subsets. In Proceedings of the Polish Symposium on Interval and Fuzzy Mathematics, Poznan, Poland, 26–29 August 1983; Albrycht, J., Wiśniewski, H., Eds.; Wydaw. Politechniki Poznańskiej: Poznan, Poland, 1985; pp. 193–195. [Google Scholar]
- Piasecki, K. Fuzzy partitions of sets. BUSEFAL
**1986**, 25, 52–60. [Google Scholar] - Riečan, B.; Mundici, D. Probability on MV-algebras. In Handbook of Measure Theory; Pap, E., Ed.; Elsevier: Amsterdam, The Netherlands, 2002; pp. 869–910. [Google Scholar]
- Di Nola, A.; Dvurečenskij, A. Product MV-algebras. Multiple-Valued Log.
**2001**, 6, 193–215. [Google Scholar] - Kôpka, F.; Chovanec, F. D-posets. Math. Slovaca
**1994**, 44, 21–34. [Google Scholar] - Frič, R. On D-posets of fuzzy sets. Math. Slovaca
**2014**, 64, 545–554. [Google Scholar] [CrossRef] - Foulis, D.J.; Bennett, M.К. Effect algebras and unsharp quantum logics. Found. Phys.
**1994**, 24, 1325–1346. [Google Scholar] [CrossRef] - Dvurečenskij, A.; Pulmannová, S. New Trends in Quantum Structures; Kluwer Academic Publishers: Dordrecht, The Netherlands, 2000. [Google Scholar]
- Atanassov, K. Intuitionistic Fuzzy Sets: Theory and Applications; Physica Verlag: New York, NY, USA, 1999. [Google Scholar]
- Atanassov, K. Intuitionistic fuzzy sets. Fuzzy Sets Syst.
**1986**, 20, 87–96. [Google Scholar] [CrossRef] - Atanassov, K. More on intuitionistic fuzzy sets. Fuzzy Sets Syst.
**1989**, 33, 37–45. [Google Scholar] [CrossRef] - Atanassov, K.; Riečan, B. On two operations over intuitionistic fuzzy sets. J. Appl. Math. Stat. Inform.
**2006**, 2, 145–148. [Google Scholar] [CrossRef] - Riečan, B. Algebraic and Proof-Theoretic Aspects of Non-Classical Logics: Papers in Honor of Daniele Mundici on the Occasion of His 60th Birthday; Springer: Berlin/Heidelberg, Germany, 2007; pp. 290–308. [Google Scholar]
- Petrovičová, J. On the entropy of partitions in product MV-algebras. Soft Comput.
**2000**, 4, 41–44. [Google Scholar] [CrossRef] - Petrovičová, J. On the entropy of dynamical systems in product MV-algebras. Fuzzy Sets Syst.
**2001**, 121, 347–351. [Google Scholar] [CrossRef] - Riečan, B. Kolmogorov–Sinaj entropy on MV-algebras. Int. J. Theor. Phys.
**2005**, 44, 1041–1052. [Google Scholar] [CrossRef] - Ďurica, M. Entropy on IF-events. Notes Intuit. Fuzzy Sets
**2007**, 13, 30–40. [Google Scholar] - Di Nola, A.; Dvurečenskij, A.; Hyčko, M.; Manara, C. Entropy on Effect Algebras with the Riesz Decomposition Property I: Basic Properties. Kybernetika
**2005**, 41, 143–160. [Google Scholar] - Di Nola, A.; Dvurečenskij, A.; Hyčko, M.; Manara, C. Entropy on Effect Algebras with the Riesz Decomposition Property II: MV-Algebras. Kybernetika
**2005**, 41, 161–176. [Google Scholar] - Giski, Z.E.; Ebrahimi, M. Entropy of Countable Partitions on Effect Algebras with the Riesz Decomposition Property and Weak Sequential Effect Algebras. Cankaya Univ. J. Sci. Eng.
**2015**, 12, 20–39. [Google Scholar] - Farnoosh, R.; Rahimi, M.; Kumar, P. Removing noise in a digital image using a new entropy method based on intuitionistic fuzzy sets. In Proceedings of the International Conference on Fuzzy Systems, Vancouver, BC, Canada, 24–29 July 2016.
- Burillo, P.; Bustince, H. Entropy on intuitionistic fuzzy sets and on interval-valued fuzzy sets. Fuzzy Sets Syst.
**1996**, 78, 305–316. [Google Scholar] [CrossRef] - Szmidt, E.; Kacprzyk, J. Entropy for intuitionistic fuzzy sets. Fuzzy Sets Syst.
**2001**, 118, 467–477. [Google Scholar] [CrossRef] - Ebrahimzadeh, A.; Giski, Z.E.; Markechová, D. Logical Entropy on Effect Algebras with the Riesz Decomposition Property. Commun. Theor. Phys.
**2017**. under review. [Google Scholar]

© 2017 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license ( http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Markechová, D. Kullback-Leibler Divergence and Mutual Information of Experiments in the Fuzzy Case. *Axioms* **2017**, *6*, 5.
https://doi.org/10.3390/axioms6010005

**AMA Style**

Markechová D. Kullback-Leibler Divergence and Mutual Information of Experiments in the Fuzzy Case. *Axioms*. 2017; 6(1):5.
https://doi.org/10.3390/axioms6010005

**Chicago/Turabian Style**

Markechová, Dagmar. 2017. "Kullback-Leibler Divergence and Mutual Information of Experiments in the Fuzzy Case" *Axioms* 6, no. 1: 5.
https://doi.org/10.3390/axioms6010005