Kullback-Leibler Divergence and Mutual Information of Experiments in the Fuzzy Case

Abstract

The main aim of this contribution is to define the notions of Kullback-Leibler divergence and conditional mutual information in fuzzy probability spaces and to derive the basic properties of the suggested measures. In particular, chain rules for mutual information of fuzzy partitions and for Kullback-Leibler divergence with respect to fuzzy P-measures are established. In addition, a convexity of Kullback-Leibler divergence and mutual information with respect to fuzzy P-measures is studied.

1. Introduction

The notion of $\sigma -$algebra S of random events and the concept of probability space $(\Omega , S, P)$ are a basis of the classical Kolmogorov probability theory [1]. A probability P is a normalized measure defined on the $\sigma -$algebra S. The event in classical probability theory is understood as an exactly and clearly defined phenomenon and, from a mathematical point of view, it is a classical, ordinary set. Consider a finite measurable partition $A$ of $(\Omega , S, P)$ with probabilities $p_1,\dots ,p_n$ of the corresponding elements of $A$. We recall that the Shannon entropy [2] of $A$ is the number $H(A)=-{\displaystyle {\sum }_{i=1}^{n}F\left(p_i\right)}$, where the function $F: [0, \infty )\to \Re$ is defined by $F(x)=-x\log x$ if $x>0$ and $F(0)=0$. In [3], we have generalized this notion to situations when the considered probability space is a fuzzy probability space $(\Omega , M, \mu )$ defined by Piasecki [4]. Instead of probability P it is considered a fuzzy P-measure $\mu$ defined on a fuzzy $\sigma -$algebra M of fuzzy subsets of $\Omega$. Recall that by a fuzzy subset f of a non-empty set $\Omega$ we mean a mapping $f:\Omega \to \left[0, 1\right]$ (Zadeh [5]). A fuzzy subset from the fuzzy $\sigma -$algebra M is interpreted as a fuzzy event; the value $\mu (f)$ is interpreted as a probability of fuzzy event $f$. The structure $(\Omega , M, \mu )$ can serve as an alternative mathematical model of probability theory for the case where the observed events are described unclearly, vaguely, so-called fuzzy events. In [6], the mutual information of fuzzy partitions of a given fuzzy probability space $(\Omega , M, \mu )$ has been defined. It was shown that the entropy of fuzzy partitions introduced and studied by the author in [3] (see also [7]) can be considered as a special case of their mutual information. The proposed measures are fuzzy analogies of the relevant terms of the classical theory and they can be used whenever it is necessary to know the quantity of information received by the realization of experiments whose results are fuzzy events. Note that in [3] (see also [8]) using the concept of entropy of fuzzy partitions we define the entropy of the fuzzy dynamical system $(\Omega , M, \mu , U)$ (where $(\Omega , M, \mu )$ is a fuzzy probability space and $U:M\to M$ is a measure $\mu$ preserving $\sigma -$homomorphism). Analogies of our results for the case of logical entropy (see, e.g., [9]) are provided in our recently published papers [10,11]. Recall that the logical entropy of the probability distribution $P=(p_1,\dots ,p_n)\in {\Re }^n$ is defined as the number $h(P)={\displaystyle \sum _{i=1}^n{p}_i(1-p_i)}$. In [9] the author deals with historical aspects of the logical entropy formula $h(P)$ and investigates the relationship between the logical entropy and Shannon’s entropy. It should be noted that other fuzzy analogies of entropy are presented in [12,13,14,15,16,17,18,19,20,21,22,23,24,25]. It is known that there are many possibilities for defining operations with fuzzy sets; an overview can be found in [26]. While our approach is based on Zadeh’s connectives [5], the authors of the cited papers used other connectives to define the fuzzy set operations.

In classical information theory [27] the mutual information is a special case of a more general quantity called Kullback-Leibler divergence, which was originally introduced by Kullback and Leibler in 1951 [28] as the divergence between two probability distributions. It is discussed in Kullback’s historic text [29]. The Kullback-Leibler divergence is also called by many other different names, as K-L divergence, whether relative entropy, or information gain. It plays an important role, as a mathematical tool, in the stability analysis of master equations [30] and Fokker-Planck equations [31], and in isothermal equilibrium fluctuations and transient nonequilibrium deviations [32] (see also [31,33]).

The main aim of this contribution is to define, using our previous results on this issue, the notion of Kullback-Leibler divergence and conditional mutual information in fuzzy probability spaces and to study properties of the suggested measures. The paper is organized as follows. In the next section, we give the basic definitions and some known facts used in this paper. Our results are presented in Section 3 and Section 4. In Section 3 we extend our study concerning the mutual information of fuzzy partitions. The notion of conditional mutual information of fuzzy partitions is introduced and subsequently chain rules for mutual information of fuzzy partitions are established. We also derive some more properties of this measure, e.g., data processing inequality. In Section 4 we define the Kullback-Leibler divergence and the conditional Kullback-Leibler divergence in fuzzy probability spaces. The basic properties of these measures are proved. Our results are summarized in Section 5.

2. Basic Definitions and Facts

We start by recalling some definitions and some known results which will be used in this contribution.

A fuzzy measurable space (Dvurečenskij [34]) is a couple $(\Omega , M)$, where $\Omega$ is a non-empty set, and M is a fuzzy $\sigma -$algebra of fuzzy subsets of $\Omega$, i.e., $M\subset {\left[0, 1\right]}^{\Omega }$ containing $1_{\Omega }$, excluding ${\left(1/2\right)}_{\Omega }$, closed under the operation $\perp :f\mapsto 1_{\Omega }-f$ (i.e., if $f\in M$, then $f^{\perp }:=1_{\Omega }-f\in M$) and countable supremums (i.e., satisfying the implication if $f_n\in M, n=1,2,\dots$, then ${\cup }_{n=1}^{\infty }{f}_n:={\sup }_n f_n\in M$). A fuzzy probability space (Piasecki [4]) is a triplet $(\Omega , M, \mu )$, where $(\Omega , M)$ is a fuzzy measurable space and the mapping $\mu :M\to \left[0, \infty \right)$ satisfies the following two conditions: (i) $\mu (f\cup f^{\perp })=1$ for all $f\in M;$ (ii) if ${\left\{f_n\right\}}_{n=1}^{\infty }$ is a sequence of pairwise W-separated fuzzy subsets from M (i.e., $f_i\le f_j^{\perp }$ (point wisely) whenever $i\ne j$), then $\mu ({\cup }_{n=1}^{\infty }{f}_n)$$={\displaystyle {\sum }_{n=1}^{\infty }\mu (f_n)}$.

The symbols ${\cup }_{n=1}^{\infty }{f}_n={\sup }_n f_n$ and ${\cap }_{n=1}^{\infty }{f}_n: ={\inf }_n f_n$ denote the fuzzy union and the fuzzy intersection of a sequence ${\left\{f_n\right\}}_{n=1}^{\infty }\subset M$, respectively, in the sense of Zadeh [5]. The complement of fuzzy subset f of $\Omega$ is a fuzzy set $f^{\perp }$ defined by $f^{\perp }(\omega )=1-f(\omega ), \omega \in \Omega$. The following notions were defined by Piasecki in [35]. A fuzzy set $f\in M$ such that $f\ge f^{\perp }$ is called a W-universum; a fuzzy set $f\in M$ such that $f\le f^{\perp }$ is called a W-empty fuzzy set. It can be proved that a fuzzy event $f\in M$ is a W-universum if and only if there exists a fuzzy event $g\in M$ such that $f=g\cup g^{\perp }$. A W-universum is interpreted as a certain event and a W-empty set as an impossible event. W-separated fuzzy events are interpreted as mutually exclusive events. Each mapping $\mu :M\to \left[0, \infty \right)$ having the properties (i) and (ii) is called, in the terminology of Piasecki, a fuzzy P-measure. Any fuzzy P-measure has all properties analogous to properties of a classical probability measure; the proofs and more details can be found in [4]. The monotonicity of fuzzy P-measure $\mu$ implies that this measure transforms M into the interval $\left[0, 1\right]$. In the following we will use the following property of fuzzy P-measure $\mu$.

(2.1) Let $g\in M.$ Then $\mu (f\cap g)=\mu (f)$ for all $f\in M$ if and only if $\mu (g)=1$.

Let $g\in M$ such that $\mu (g)>0$. Then the mapping $\mu (\cdot /g):M\to \left[0, 1\right]$ defined by the formula

$$\mu (f/g)=\frac{\mu (f\cap g)}{\mu (g)}, f\in M,$$

is a fuzzy P-measure on M; it is called a conditional probability.

Definition 1 

[36]. By a fuzzy partition (of a space $(\Omega , M, \mu )$) we understand a finite collection $\xi =\left\{f_1,\dots ,f_n\right\}$ of pairwise W-separated fuzzy subsets from M such that $\mu ({\cup }_{i=1}^n{f}_i)=1$.

Every fuzzy partition $\xi =\left\{f_1,\dots ,f_n\right\}$ of $(\Omega , M, \mu )$ represents in the sense of the classical probability theory a random experiment with a finite number of outcomes $f_i, i=1,2,\dots ,n$ (which are fuzzy events) with a probability distribution $p_i=\mu (f_i)$, $i=1,2,\dots ,n$, since $p_i\ge 0$ for $i=1,2,\dots ,n$, and ${\displaystyle \sum _{i=1}^n{p}_i}=$ ${\displaystyle \sum _{i=1}^n\mu (f_i})$ $=\mu ({\cup }_{i=1}^n{f}_i)=1$. For that reason, we have defined in [3] the entropy of $\xi =\left\{f_1,\dots ,f_n\right\}$ by Shannon’s formula:

$$H_{\mu }(\xi )=-{\displaystyle {\sum }_{i=1}^{n}F(\mu (f_i)}),$$

(1)

where:

$$F: [0,\infty )\to \Re , F(x)= \{\begin{array}{ll}x\log x,\hfill & \mathrm{if} x>0;\hfill \\ 0,\hfill & \mathrm{if} x=0.\hfill \end{array}$$

In accordance with the classical theory the log is to the base 2 and entropy is expressed in bits.

Definition 2 

[6]. Let $\xi =\left\{f_1,\dots ,f_n\right\}$ and $\eta =\left\{g_1,\dots ,g_m\right\}$ be two fuzzy partitions of a fuzzy probability space $(\Omega , M, \mu ).$ A conditional entropy of $\eta$ given a fuzzy event $f_i\in \xi$ is defined by:

$$H_{\mu }(\eta /f_i)=-{\displaystyle {\sum }_{j=1}^{m}F(\dot{\mu }(g_j/f_i))},$$

where:

$$\dot{\mu }(g_j/f_i)= \{\begin{array}{ll}\mu (g_j/f_i),\hfill & if \mu (f_i)>0;\hfill \\ 0,\hfill & if \mu (f_i)=0.\hfill \end{array}$$

A conditional entropy of $\eta$ assuming a realization of the experiment $\xi$ is defined by the formula:

$$H_{\mu }(\eta /\xi )=-{\displaystyle {\sum }_{i=1}^n\mu (f_i)\cdot H_{\mu }(\eta /f_i)}=-{\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m\mu (f_i)\cdot F(\dot{\mu }(g_j/f_i))}}.$$

(2)

In the following, we will use the convention (based on continuity arguments) that $x\log \frac{x}{0}=\infty$ if $x>0$, and $0\log \frac{0}{x}=0$ if $x\ge 0$. It is easy to see that we can rewrite Equation (2) in the following way:

$$H_{\mu }(\eta /\xi )=-{\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m\mu (f_i\cap g_j)}}\cdot \log \frac{\mu (f_i\cap g_j)}{\mu (f_i)}.$$

(3)

As in [3] we define in the set of all fuzzy partitions of a fuzzy probability space $(\Omega , M, \mu )$ the relation $\prec$: Let $\xi , \eta$ be two fuzzy partitions of a fuzzy probability space $(\Omega , M, \mu )$. Then we write $\xi \prec \eta$ (and we say that the fuzzy partition $\eta$ is a refinement of the fuzzy partition $\xi$) iff for every $g\in \eta$ there exists $f\in \xi$ such that $g\le f$. Given two fuzzy partitions $\xi =\left\{f_1,\dots ,f_n\right\}$ and $\eta =\left\{g_1,\dots ,g_m\right\}$ of a fuzzy probability space $(\Omega , M, \mu )$, their common refinement $\xi \vee \eta$ is defined as the system $\xi \vee \eta =\{f_i\cap g_j; i=1,\dots ,n, j=1,\dots ,m \}$. The fuzzy partition $\xi \vee \eta$ represents a joint experiment of experiments $\xi ,\eta$. Evidently, $\xi \prec \xi \vee \eta$ and $\eta \prec \xi \vee \eta$. If ${\xi }_1, {\xi }_2,\dots ,{\xi }_n$ are fuzzy partitions of a fuzzy probability space $(\Omega , M, \mu )$, then we put ${\vee }_{i=1}^n{\xi }_i={\xi }_1\vee {\xi }_2\vee \dots \vee {\xi }_n$. Two fuzzy partitions $\xi =\left\{f_1,\dots ,f_n\right\}$ and $\eta =\left\{g_1,\dots ,g_m\right\}$ of a given fuzzy probability space $(\Omega , M, \mu )$ are called statistically independent, if $\mu (f_i\cap g_j)=\mu (f_i)\cdot \mu (g_j)$, for $i=1,2,\dots ,n$, $j=1, 2,\dots ,m$.

Example 1.

Let $\Omega =\left[0, 1\right],$ and let $f:\Omega \to \left[0, 1\right]$ be defined by $f(\omega )=\omega , \omega \in \Omega$. Consider a fuzzy measurable space $(\Omega , M)$, where $M=\left\{1_{\Omega }, 0_{\Omega },f,f^{\perp },f\cup f^{\perp },f\cap f^{\perp }\right\}$. Then it is easy to verify that the mapping $\mu :M\to \left[0, 1\right]$ defined by the equalities $\mu \left(1_{\Omega }\right)=\mu (f\cup f^{\perp })=1$, $\mu \left(0_{\Omega }\right)=\mu (f\cap f^{\perp })=0$, $\mu (f)=p$, $\mu (f^{\perp })=1-p$, where $p\in \left(0, 1\right)$, is a fuzzy P-measure and the system $(\Omega , M, \mu )$ is a fuzzy probability space. The sets $\xi =\left\{f, f^{\perp }\right\}$, $\eta =\left\{f\cup f^{\perp }\right\}$, $\varsigma =\left\{1_{\Omega }\right\}$ are fuzzy partitions of $(\Omega , M, \mu )$ such that $\varsigma \prec \eta \prec \xi$. We can calculate their entropy: $H_{\mu }(\xi )=-p\log p-(1-p)\log (1-p)$, $H_{\mu }(\eta )=H_{\mu }(\varsigma )=0$. This makes sense, because the partitions $\eta$ and $\varsigma$ represent experiments whose results are certain events. In particular, if $p=\frac{1}{2}$, then $H_{\mu }(\xi )=\log 2=1$ bit.

The entropy and the conditional entropy of fuzzy partitions of a fuzzy probability space $(\Omega , M, \mu )$ satisfy all properties analogous to properties of Shannon’s entropy of measurable partitions in the classical case; the proofs can be found in [3,6], respectively. We present some of them. If $\xi , \eta , \varsigma$ are fuzzy partitions of a fuzzy probability space $(\Omega , M,\mu )$, then:

(2.2)

$\xi \prec \eta$ implies $H_{\mu }(\xi )\le H_{\mu }(\eta );$

(2.3)

$H_{\mu }(\eta \vee \varsigma /\xi )=H_{\mu }(\varsigma /\xi \vee \eta )+H_{\mu }(\eta /\xi );$

(2.4)

$\xi \prec \eta$ implies $H_{\mu }(\xi /\varsigma )\le H_{\mu }(\eta /\varsigma );$

(2.5)

$\xi \prec \eta$ implies $H_{\mu }(\varsigma /\xi )\ge H_{\mu }(\varsigma /\eta );$

(2.6)

$H_{\mu }(\xi /\eta )\le H_{\mu }(\xi )$ with the equality if and only if $\xi , \eta$ are statistically independent;

(2.7)

$H_{\mu }(\eta \vee \varsigma /\xi )\le H_{\mu }(\eta /\xi )+H_{\mu }(\varsigma /\xi );$

(2.8)

$H_{\mu }(\xi \vee \eta )=H_{\mu }(\xi )+H_{\mu }(\eta /\xi ).$

Definition 3 

[6]. Let $\xi , \eta$ be two fuzzy partitions of a given fuzzy probability space $(\Omega , M,\mu ).$ The mutual information of $\xi$ and $\eta$ is defined by the formula:

$$I_{\mu }(\xi ,\eta )= H_{\mu }(\xi )- H_{\mu }(\xi /\eta ).$$

(4)

As a simple consequence of (2.8) we have:

$$I_{\mu }(\xi ,\eta )= H_{\mu }(\xi )+ H_{\mu }(\eta ) -H_{\mu }(\xi \vee \eta ).$$

(5)

It is evident that $I_{\mu }(\xi ,\xi )=H_{\mu }(\xi )$, and $I_{\mu }(\xi ,\eta )$ = $I_{\mu }(\eta ,\xi )$. Hence, we can write:

$$I_{\mu }(\xi ,\eta )= H_{\mu }(\eta )- H_{\mu }(\eta /\xi ).$$

(6)

The proofs of the following two theorems can be found in [6].

Theorem 1 

[6]. Let $\xi =\left\{f_1,\dots ,f_n\right\}$ and $\eta =\left\{g_1,\dots ,g_m\right\}$ be two fuzzy partitions of a fuzzy probability space $(\Omega , M, \mu ).$ Then:

$$I_{\mu }(\xi ,\eta )={\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m\mu (f_i\cap g_j)\log \frac{\mu (f_i\cap g_j)}{\mu (f_i)\cdot \mu (g_j)}}}.$$

Theorem 2 

[6]. $I_{\mu }(\xi ,\eta )\ge 0$ with the equality if and only if $\xi , \eta$ are statistically independent.

3. Mutual Information and Conditional Mutual Information in Fuzzy Probability Spaces

In this section by using our previous results we introduce the notion of conditional mutual information in fuzzy probability spaces. We derive chain rules for mutual information of fuzzy partitions and we will prove some more properties concerning these measures.

Definition 4.

Let $\xi , \eta , \varsigma$ be fuzzy partitions of a fuzzy probability space $(\Omega , M,\mu ).$ Then the conditional mutual information of $\xi$ and $\eta$ given $\varsigma$ is defined by the formula:

$$I_{\mu }(\xi ,\eta /\varsigma )=H_{\mu }(\xi /\varsigma ) -H_{\mu }(\xi /\eta \vee \varsigma ).$$

(7)

Remark 1.

Since, according to (2.5), we have $H_{\mu }(\xi /\varsigma ) \ge H_{\mu }(\xi /\eta \vee \varsigma ),$ for the conditional mutual information the inequality $I_{\mu }(\xi ,\eta /\varsigma )\ge 0$ holds.

Theorem 3.

For fuzzy partitions $\xi , \eta , \varsigma$ of a fuzzy probability space $(\Omega , M,\mu ),$ it holds:

$$I_{\mu }(\xi ,\eta \vee \varsigma )=I_{\mu }(\xi ,\eta )+I_{\mu }(\xi ,\varsigma /\eta )=I_{\mu }(\xi ,\varsigma )+I_{\mu }(\xi ,\eta /\varsigma ).$$

Proof. 

By simple calculation we obtain:

$$\begin{array}{c}{I}_{\mu }(\xi ,\eta )+I_{\mu }(\xi ,\varsigma /\eta )=H_{\mu }(\xi )-H_{\mu }(\xi /\eta )+H_{\mu }(\xi /\eta )-H_{\mu }(\xi /\eta \vee \varsigma )\\ =H_{\mu }(\xi )-H_{\mu }(\xi /\eta \vee \varsigma )=I_{\mu }(\xi ,\eta \vee \varsigma ).\end{array}$$

Analogously we obtain the second equality. □

Let $\xi , \eta$ be fuzzy partitions of a fuzzy probability space $(\Omega , M,\mu )$. Denote ${\xi }_0=\left\{1_{\Omega }\right\}$. Evidently, we have ${\xi }_0\vee \xi =\xi$, and $H_{\mu }(\xi /{\xi }_0)=H_{\mu }(\xi )$. Hence, we can write:

$$I_{\mu }(\xi ,\eta /{\xi }_0)=H_{\mu }(\xi /{\xi }_0) -H_{\mu }(\xi /\eta \vee {\xi }_0)= H_{\mu }(\xi )- H_{\mu }(\xi /\eta )=I_{\mu }(\xi ,\eta ).$$

Theorem 4 (Chain rules).

Let ${\xi }_1, {\xi }_2,\dots ,{\xi }_n$ and $\eta$ be fuzzy partitions of a fuzzy probability space $(\Omega , M,\mu ).$ Then, for $n=1,2,\dots ,$ the following equalities hold:

(i) 

$H_{\mu }({\xi }_1\vee {\xi }_2\vee \dots \vee {\xi }_n)=$${\displaystyle {\sum }_{i=1}^n{H}_{\mu }}({\xi }_i/{\vee }_{k=0}^{i-1}{\xi }_k);$

(ii) 

$H_{\mu }({\vee }_{i=1}^n{\xi }_i/\eta )$$={\displaystyle {\sum }_{i=1}^n{H}_{\mu }}({\xi }_i/({\vee }_{k=0}^{i-1}{\xi }_k)\vee \eta );$

(iii) 

$I_{\mu }({\vee }_{i=1}^n{\xi }_i,\eta )=$${\displaystyle {\sum }_{i=1}^n{I}_{\mu }}({\xi }_i,\eta /{\vee }_{k=0}^{i-1}{\xi }_k).$

Proof. 

The first two equalities may be obtained using the principle of mathematical induction from the properties (2.8) and (2.3). By Equation (4), using the two previous equalities and Equation (7), we obtain:

$$\begin{gathered} I_{\mu }({\vee }_{i=1}^n{\xi }_i,\eta )= H_{\mu }({\vee }_{i=1}^n{\xi }_i)- H_{\mu }({\vee }_{i=1}^n{\xi }_i/\eta ) \\ ={\displaystyle {\sum }_{i=1}^n{H}_{\mu }}({\xi }_i/{\vee }_{k=0}^{i-1}{\xi }_k)-{\displaystyle {\sum }_{i=1}^n{H}_{\mu }}({\xi }_i/({\vee }_{k=0}^{i-1}{\xi }_k)\vee \eta ) \\ ={\displaystyle {\sum }_{i=1}^n(H_{\mu }}({\xi }_i/{\vee }_{k=0}^{i-1}{\xi }_k) -H_{\mu }({\xi }_i/({\vee }_{k=0}^{i-1}{\xi }_k)\vee \eta ))={\displaystyle {\sum }_{i=1}^n{I}_{\mu }}({\xi }_i,\eta /{\vee }_{k=0}^{i-1}{\xi }_k). \square \end{gathered}$$

Definition 5.

Let $\xi , \eta , \varsigma$ be fuzzy partitions of a fuzzy probability space $(\Omega , M,\mu ).$ We say that $\xi$ is conditionally independent to $\varsigma$ given $\eta$ (and write $\xi \to \eta \to \varsigma$) if $I_{\mu }(\xi ,\varsigma /\eta )=0.$

Theorem 5.

For fuzzy partitions $\xi , \eta , \varsigma$ of a fuzzy probability space $(\Omega , M,\mu ),$ it holds:

$$\xi \to \eta \to \varsigma \mathit{if} \mathit{and} \mathit{only} \mathit{if} \varsigma \to \eta \to \xi .$$

Proof. 

Let $\xi \to \eta \to \varsigma .$ Then:

$$0=I_{\mu }(\xi ,\varsigma /\eta )=H_{\mu }(\xi /\eta )-H_{\mu }(\xi /\eta \vee \varsigma ),$$

and according to the property (2.8) we can write:

$$H_{\mu }(\xi /\eta )=H_{\mu }(\xi /\eta \vee \varsigma )=H_{\mu }(\xi \vee \eta \vee \varsigma )-H_{\mu }(\eta \vee \varsigma ).$$

Calculate:

$$\begin{gathered} I_{\mu }(\varsigma ,\xi /\eta )=H_{\mu }(\varsigma /\eta )-H_{\mu }(\varsigma /\xi \vee \eta ) \\ =H_{\mu }(\varsigma \vee \eta )-H_{\mu }(\eta )-H_{\mu }(\xi \vee \eta \vee \varsigma )+ H_{\mu }(\xi \vee \eta ) \\ =H_{\mu }(\xi \vee \eta )- H_{\mu }(\eta )- H_{\mu }(\xi /\eta )= H_{\mu }(\xi /\eta )- H_{\mu }(\xi /\eta )=0. \end{gathered}$$

It follows that $\varsigma \to \eta \to \xi$. The opposite implication is obvious. □

According to Theorem 5, we may say that $\xi$ and $\varsigma$ are conditionally independent given $\eta$ and write $\xi \leftrightarrow \eta \leftrightarrow \varsigma$ instead of $\xi \to \eta \to \varsigma$.

Theorem 6.

For fuzzy partitions $\xi , \eta , \varsigma$ of a fuzzy probability space $(\Omega , M,\mu )$ such that $\xi \to \eta \to \varsigma ,$ we have:

(i) 

$I_{\mu }(\xi \vee \eta ,\varsigma )=$$I_{\mu }(\eta ,\varsigma );$

(ii) 

$I_{\mu }(\eta ,\varsigma )=$$I_{\mu }(\xi ,\varsigma )+$$I_{\mu }(\varsigma ,\eta /\xi );$

(iii) 

$I_{\mu }(\xi ,\eta /\varsigma )\le$$I_{\mu }(\xi ,\eta );$

(iv) 

$I_{\mu }(\xi ,\eta )\ge I_{\mu }(\xi ,\varsigma ).$

Proof. 

(i)

Since by the assumption $I_{\mu }(\xi ,\varsigma /\eta )=0,$ using the chain rule for logical mutual information, we obtain:

$$I_{\mu }(\xi \vee \eta ,\varsigma )=I_{\mu }(\eta \vee \xi ,\varsigma )=I_{\mu }(\eta ,\varsigma )+I_{\mu }(\xi ,\varsigma /\eta )=I_{\mu }(\eta ,\varsigma ).$$

(ii)

According to Theorem 3 we have $I_{\mu }(\xi \vee \eta ,\varsigma )=$$I_{\mu }(\varsigma ,\xi )+$$I_{\mu }(\varsigma ,\eta /\xi )$. Hence, using the equality (i) of this theorem, we obtain:

$$I_{\mu }(\eta ,\varsigma )=I_{\mu }(\xi \vee \eta ,\varsigma )=I_{\mu }(\varsigma ,\xi )+I_{\mu }(\varsigma ,\eta /\xi ).$$

(iii)

Since $I_{\mu }(\xi ,\eta )\ge 0,$ from (ii) it follows the inequality:

$$I_{\mu }(\varsigma ,\eta /\xi )\le I_{\mu }(\varsigma ,\eta ).$$

By Theorem 5 we can interchange $\xi$ and $\varsigma$. Doing so we obtain $I_{\mu }(\xi ,\eta /\varsigma )\le$$I_{\mu }(\xi ,\eta )$.

(iv)

By Theorem 3, the mutual information $I_{\mu }(\xi ,\eta \vee \varsigma )$ can be expressed in two different ways:

$$\begin{array}{c}{I}_{\mu }(\xi ,\eta \vee \varsigma )=I_{\mu }(\xi ,\eta )+I_{\mu }(\xi ,\varsigma /\eta )\\ =I_{\mu }(\xi ,\varsigma ) +I_{\mu }(\xi ,\eta /\varsigma ).\end{array}$$

Since $\xi \to \eta \to \varsigma$, we have $I_{\mu }(\xi ,\varsigma /\eta )=0$, and, therefore, it holds $I_{\mu }(\xi ,\eta )=I_{\mu }(\xi ,\eta \vee \varsigma )$. Using the second equality, we obtain:

$$I_{\mu }(\xi ,\eta )=I_{\mu }(\xi ,\eta \vee \varsigma )=I_{\mu }(\xi ,\varsigma ) +I_{\mu }(\xi ,\eta /\varsigma ).$$

Since $I_{\mu }(\xi ,\eta /\varsigma )\ge 0$, we have $I_{\mu }(\xi ,\eta )\ge I_{\mu }(\xi ,\varsigma )$. □

Note that, in the classical theory, the last assertion from the previous theorem is known as the data processing inequality.

4. Kullback-Leibler Divergence with Respect to Fuzzy P-Measures

In this part we define the Kullback-Leibler divergence in fuzzy probability spaces and its conditional version. We prove basic properties of these measures; in particular, Gibb’s inequality. Finally, using the concept of conditional Kullback-Leibler divergence we establish a chain rule for Kullback-Leibler divergence with respect to fuzzy P-measures. In the proofs we use the following known log-sum inequality: for non-negative real numbers $a_1, a_2,\dots ,a_n$, $b_1, b_2,\dots ,b_n$, it holds:

$${\displaystyle \sum _{i=1}^n{a}_i\log \frac{a_i}{b_i}}\ge ({\displaystyle \sum _{i=1}^n{a}_i})\log \frac{{\displaystyle {\sum }_{i=1}^n{a}_i}}{{\displaystyle {\sum }_{i=1}^n{b}_i}}$$

(8)

with the equality if and only if $\frac{a_i}{b_i}$ is constant. Recall that we use the convention that $x\log \frac{x}{0}=\infty$ if $x>0$, and $0\log \frac{0}{x}=0$ if $x\ge 0$.

Definition 6.

Let $\mu ,$ $\nu$ be fuzzy P-measures on a common fuzzy measurable space $(\Omega , M).$ Then, for any fuzzy partition $\xi$ of fuzzy probability spaces $(\Omega , M,\mu ),$ $(\Omega , M,\nu ),$ we define the Kullback-Leibler divergence $D_{\xi }(\mu$ ‖ $\nu )$ by:

$$D_{\xi }(\mu \Vert \nu )={\displaystyle {\sum }_{i=1}^n\mu (f_i}) \log \frac{\mu (f_i)}{\nu (f_i)}.$$

Remark 2.

The Kullback-Leibler divergence is not a metric in a true sense since it is not symmetric, i.e., the equality $D_{\xi }(\mu$ ‖ $\nu )$$=D_{\xi }(\nu$ ‖ $\mu )$ is not necessarily true (as shown in the example that follows), and does not satisfy the triangle inequality.

Example 2.

Consider the fuzzy measurable space $(\Omega , M)$ from Example 1 and the following two fuzzy P-measures $\mu , \nu$ defined on M: $\mu$ is defined as in Example 1 and $\nu$ is defined in a similar way: $\nu \left(1_{\Omega }\right)=\nu (f\cup f^{\perp })=1,$ $\nu \left(0_{\Omega }\right)=\nu (f\cap f^{\perp })=0,$ $\nu (f)=q,$ $\nu (f^{\perp })=1-q,$ where $q\in \left(0, 1\right).$ Then, for the fuzzy partition $\xi =\left\{f, f^{\perp }\right\},$ we obtain:

$$D_{\xi }(\mu \Vert \nu )=p\cdot \log \frac{p}{q}+\left(1-p\right)\cdot \log \frac{1-p}{1-q}, \mathit{and} D_{\xi }(\nu \Vert \mu )=q\cdot \log \frac{q}{p}+\left(1-q\right)\cdot \log \frac{1-q}{1-p}.$$

If $p=q,$ then $D_{\xi }(\mu$ ‖ $\nu )=$$D_{\xi }(\nu$ ‖ $\mu )=0.$ If $p=\frac{1}{2},$ $q=\frac{1}{3},$ then we have:

$$D_{\xi }(\mu \Vert \nu )=\frac{1}{2}\cdot \log \frac{\frac{1}{2}}{\frac{1}{3}}+\frac{1}{2}\cdot \log \frac{\frac{1}{2}}{\frac{2}{3}}=\frac{1}{2}\cdot \log \frac{3}{2}+\frac{1}{2}\cdot \log \frac{3}{4}=0.084963 \mathit{bit}$$

and:

$$D_{\xi }(\nu \Vert \mu )=\frac{1}{3}\cdot \log \frac{\frac{1}{3}}{\frac{1}{2}}+\frac{2}{3}\cdot \log \frac{\frac{2}{3}}{\frac{1}{2}}=\frac{1}{3}\cdot \log \frac{2}{3}+\frac{2}{3}\cdot \log \frac{4}{3}=0.0817036 \mathit{bit}.$$

This means that $D_{\xi }(\mu$ ‖ $\nu )\ne$$D_{\xi }(\nu$ ‖ $\mu ),$ in general.

The following result suggests interpretation of Kullback-Leibler divergence as a measure of how different two fuzzy P-measures (over the same fuzzy partition) are.

Theorem 7.

Let $\xi =\left\{f_1,\dots ,f_n\right\}$ be a fuzzy partition of fuzzy probability spaces $(\Omega , M,\mu )$, $(\Omega , M,\nu )$. Then $D_{\xi }(\mu$ ‖ $\nu )\ge 0$ (Gibb’s inequality) with the equality if and only if $\mu (f_i)=\nu (f_i)$, for $i=1, 2,\dots ,n$.

Proof. 

Putting $a_i=\mu (f_i)$ and $b_i=\nu (f_i)$, for $i=1, 2,\dots ,n$, in the log-sum inequality, we obtain ${\displaystyle {\sum }_{i=1}^n{a}_i}={\displaystyle {\sum }_{i=1}^n\mu (f_i)}=\mu ({\cup }_{i=1}^n{f}_i)=1;$ analogously we obtain ${\displaystyle {\sum }_{i=1}^n{b}_i}=1$. Therefore, by Equation (8):

$$\begin{gathered} D_{\xi }(\mu \Vert \nu )={\displaystyle {\sum }_{i=1}^n\mu (f_i)} \log \frac{\mu (f_i)}{\nu (f_i)} \\ \ge \left({\displaystyle {\sum }_{i=1}^n\mu (f_i)}\right) \log \frac{{\displaystyle {\sum }_{i=1}^n\mu (f_i)}}{{\displaystyle {\sum }_{i=1}^n\nu (f_i)}}=1\cdot \log \frac{1}{1}=0 \end{gathered}$$

with the equality if and only if $\frac{\mu (f_i)}{\nu (f_i)}=\alpha$ for $i=1, 2,\dots ,n$, where $\alpha$ is constant. Taking the sum for all $i=1, 2,\dots ,n$, we obtain $\alpha =1$. This means that $D_{\xi }(\mu$ ‖ $\nu )=0$ if and only if $\mu (f_i)=\nu (f_i)$, for $i=1, 2,\dots ,n$. □

Theorem 8.

Let $\mu ,$ $\nu$ be fuzzy P-measures on a common fuzzy measurable space $(\Omega , M).$ Then, for any fuzzy partition $\xi =\left\{f_1,\dots ,f_n\right\}$ of fuzzy probability spaces $(\Omega , M,\mu ),$ $(\Omega , M,\nu ),$ it holds:

$$H_{\mu }(\xi )=\log n-D_{\xi }(\mu \Vert \nu ),$$

where n is the number of members of $\xi$, and $\nu$ is the uniform probability distribution over $\xi$, i.e., $\nu (f_i)=\frac{1}{n}$, for $i=1, 2,\dots ,n$.

Proof. 

Calculate:

$$\begin{gathered} D_{\xi }(\mu \Vert \nu )={\displaystyle {\sum }_{i=1}^n\mu (f_i)} \log \frac{\mu (f_i)}{\nu (f_i)}={\displaystyle {\sum }_{i=1}^n\mu (f_i)} \log \frac{\mu (f_i)}{\frac{1}{n}} \\ ={\displaystyle {\sum }_{i=1}^n\mu (f_i)} \left(\log \mu (f_i)-\log n^{-1}\right) \\ ={\displaystyle {\sum }_{i=1}^n\mu (f_i)} \log \mu (f_i) +\log n=\log n-H_{\mu }(\xi ). \square \end{gathered}$$

As a consequence of the previous two theorems we obtain the following property of entropy of fuzzy partitions:

Corollary 1.

For any fuzzy partition $\xi$ of a fuzzy probability space $(\Omega , M,\mu ),$ it holds $H_{\mu }(\xi )\le \log n,$ where n denotes the number of members of $\xi ,$ with the equality if and only if $\mu$ is the uniform probability distribution over $\xi .$

Proof. 

Let $\nu$ be the uniform probability distribution over $\xi =\left\{f_1,\dots ,f_n\right\}$. Then, according to the previous theorem and Gibb’s inequality (Theorem 7) we have:

$$0\le D_{\xi }(\mu \Vert \nu )=\log n-H_{\mu }(\xi ),$$

which implies the inequality $H_{\mu }(\xi )\le \log n$, where n is the number of members of $\xi$. Since by Theorem 7 $D_{\xi }(\mu$ ‖ $\nu )=0$ if and only if $\mu (f_i)=\nu (f_i)$, for $i=1, 2,\dots ,n$, the equality $H_{\mu }(\xi )=\log n$ holds if and only if $\mu$ is the uniform probability distribution over $\xi$. □

In the following, a concavity of entropy $H_{\mu }(\xi )$ as a function of $\mu$ and convexity of K-L divergence with respect to fuzzy P-measures are shown. We recall, for the convenience of the reader, the definitions of convex and concave function:

A real-valued function $f$ is said to be convex over an interval $\left[a,b\right]$ if for every $x_1,x_2\in \left[a,b\right]$ and for any real number $\alpha \in \left[0, 1\right]$,

$$f \left(\alpha x_1+(1-\alpha )x_2\right)\le \alpha f\left(x_1\right)+(1-\alpha )f\left(x_2\right).$$

A real-valued function $f$ is said to be concave over an interval $\left[a,b\right]$ if for every $x_1,x_2\in \left[a,b\right]$ and for any real number $\alpha \in \left[0, 1\right]$,

$$f \left(\alpha x_1+(1-\alpha )x_2\right)\ge \alpha f\left(x_1\right)+(1-\alpha )f\left(x_2\right).$$

Remark 3.

In the proofs of some of the next assertions, the following known properties are used:

(i) 

A function $f$ is concave over an interval if and only if the function $-f$ is convex over the interval.

(ii) 

The sum of two concave functions is itself concave; the sum of two convex functions is itself convex.

(iii) 

Every real-valued affine function, i.e., each function of the form $f(x)=ax+b,$ $a,b\in \Re ,$ is simultaneously convex and concave.

It is easy to prove the following proposition.

Proposition 1.

Let ${\mu }_1,$ ${\mu }_2$ be fuzzy P-measures on a given fuzzy measurable space $(\Omega , M).$ Then, for every real number $\alpha \in \left[0, 1\right],$ the mapping $\alpha {\mu }_1+(1-\alpha ){\mu }_2$ is a fuzzy P-measure on $(\Omega , M).$

Theorem 9 (Concavity of entropy).

Let $\xi$ be a fuzzy partition of fuzzy probability spaces $(\Omega , M,{\mu }_1),$ $(\Omega , M,{\mu }_2).$ Then, for any real number $\alpha \in \left[0, 1\right],$ the following inequality holds:

$$\alpha H_{{\mu }_1}(\xi )+\left(1-\alpha \right) H_{{\mu }_2}(\xi )\le H_{\alpha {\mu }_1+(1-\alpha ){\mu }_2}(\xi ).$$

Proof. 

Assume that $\xi =\left\{f_1,f_2,\dots ,f_n\right\}.$ Since the function F is convex, we obtain:

$$\begin{gathered} \alpha H_{{\mu }_1}(\xi )+\left(1-\alpha \right) H_{{\mu }_2}(\xi )=- \alpha {\displaystyle \sum _{i=1}^{n}F\left({\mu }_1\left(f_i\right)\right)}-\left(1-\alpha \right) {\displaystyle \sum _{i=1}^{n}F\left({\mu }_2\left(f_i\right)\right)} \\ =- {\displaystyle \sum _{i=1}^n(\alpha F\left({\mu }_1\left(f_i\right)\right) }+\left(1-\alpha \right) F\left({\mu }_2\left(f_i\right)\right))\le - {\displaystyle \sum _{i=1}^n }F\left(\alpha {\mu }_1\left(f_i\right)+\left(1-\alpha \right){\mu }_2\left(f_i\right)\right) \\ =- {\displaystyle \sum _{i=1}^n }F\left(\left(\alpha {\mu }_1+\left(1-\alpha \right){\mu }_2\right)\left(f_i\right)\right)=H_{\alpha {\mu }_1+(1-\alpha ){\mu }_2}(\xi ), \end{gathered}$$

which proves the concavity of entropy $H_{\mu }(\xi )$ as a function of fuzzy P-measure $\mu$. □

Theorem 10 (Convexity of K-L divergence).

The K-L divergence $D_{\xi }(\mu$ ‖ $\nu )$ is convex in the pair $(\mu , \nu ),$ i.e., for fuzzy P- measures ${\mu }_1,$ ${\nu }_1$ and ${\mu }_2,$ ${\nu }_2$ on a common fuzzy measurable space $(\Omega , M),$ we have, for any fuzzy partition $\xi$ of fuzzy probability spaces $(\Omega , M,{\mu }_i),$ $(\Omega , M,{\nu }_i), i=1,2,$ and for any real number $\alpha \in \left[0, 1\right],$ the following inequality:

$$D_{\xi }(\alpha {\mu }_1+(1-\alpha ){\mu }_2\Vert \alpha {\nu }_1+(1-\alpha ){\nu }_2)\le \alpha D_{\xi }({\mu }_1\Vert {\nu }_1)+(1-\alpha )D_{\xi }({\mu }_2\Vert {\nu }_2).$$

Proof. 

Let $\xi =\left\{f_1,\dots ,f_n\right\}$ be a fuzzy partition of fuzzy probability spaces $(\Omega , M, {\mu }_1)$, $(\Omega , M, {\mu }_2)$, $(\Omega , M, {\nu }_1)$, $(\Omega , M, {\nu }_2)$. Fix $i\in \left\{1, 2,\dots ,n\right\}$. Putting $a_1=\alpha {\mu }_1(f_i)$, $a_2=(1-\alpha ){\mu }_2(f_i)$, $b_1=\alpha {\nu }_1(f_i)$, $b_2=(1-\alpha ){\nu }_2(f_i)$ in the log-sum inequality, we obtain:

$$\begin{gathered} (\alpha {\mu }_1(f_i) +(1-\alpha ){\mu }_2(f_i)) \cdot \log \frac{\alpha {\mu }_1(f_i)+(1-\alpha ){\mu }_2(f_i)}{\alpha {\nu }_1(f_i)+(1-\alpha ){\nu }_2(f_i)} \\ \le \alpha {\mu }_1(f_i) \cdot \log \frac{\alpha {\mu }_1(f_i)}{\alpha {\nu }_1(f_i)}+(1-\alpha ){\mu }_2(f_i)\cdot \log \frac{(1-\alpha ){\mu }_2(f_i)}{(1-\alpha ){\nu }_2(f_i).} \end{gathered}$$

Summing these inequalities over all i, we obtain what was claimed. □

Theorem 11.

Let $\xi =\left\{f_1,\dots ,f_n\right\}$ and $\eta =\left\{g_1,\dots ,g_m\right\}$ be two fuzzy partitions of a fuzzy probability space $(\Omega , M, \mu ).$ Then the mutual information $I_{\mu }(\xi ,\eta )$ of $\xi$ and $\eta$ is a concave function of $\mu$ for fixed $\mu (g_j/f_i),$ $j=1, 2,\dots ,m,$ $i=1,\dots ,n.$ The mutual information $I_{\mu }(\xi ,\eta )$ of $\xi$ and $\eta$ is a convex function of $\mu (g_j/f_i),$ for fixed $\mu (f_i), i=1,\dots ,n.$

Proof. 

Let us prove the first assertion. By Equations (6) and (2) we have:

$$I_{\mu }(\xi ,\eta )=H_{\mu }(\eta )-{\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m\mu (f_i)\cdot F(\dot{\mu }(g_j/f_i))}}.$$

(9)

Let $g_j\in M$. Since $f_k\cap g_j\le f_k\le f_l^{\perp }\le f_l^{\perp }\cup g_j^{\perp }={(f_l\cap g_j)}^{\perp }$ whenever $k\ne l$, the system $\{f_1\cap g_j,\dots ,f_n\cap g_j\}$ is a system of pairwise W-separated fuzzy subsets from M. Due to the assumption $\mu ({\cup }_{i=1}^n{f}_i)=1$, the property (2.1) and additivity of the fuzzy P-measure, we get:

$$\mu (g_j)=\mu \left(({\cup }_{i=1}^n{f}_i)\cap g_j\right)=\mu \left({\cup }_{i=1}^n(f_i\cap g_j)\right)={\displaystyle \sum _{i=1}^n\mu (f_i\cap g_j)}={\displaystyle \sum _{i:\mu (f_i)>0}\mu (f_i)\mu (g_j/f_i)}.$$

This means that, for fixed $\mu (g_j/f_i)$, $\mu (g_j)$ is a linear function of $\mu (f_i)$. Therefore, $H_{\mu }(\eta )$, which is a concave function of $\mu (g_j)$, is a concave function of $\mu (f_i)$. The second term of the difference in Equation (9) is a linear function of $\mu (f_i)$. Hence, (see Remark 3) the difference in Equation (9) is a concave function of $\mu (f_i)$. Thus, the first part of theorem is proved.

Now, let us prove the second part. We fix $\mu$ over $\xi$ and consider two different conditional fuzzy P-measures ${\mu }_1(g_j/f_i)$ and ${\mu }_2(g_j/f_i)$, defined for $f_i\in \xi$, $\mu (f_i)>0$, and $j=1, 2,\dots ,m$. Then, for $i=1,\dots ,n$, $j=1, 2,\dots ,m$, we have:

$${\mu }_1(f_i\cap g_j)=\mu (f_i) {\dot{\mu }}_1(g_j/f_i), \mathrm{and} {\mu }_2(f_i\cap g_j)=\mu (f_i) {\dot{\mu }}_2(g_j/f_i).$$

According to Proposition 1 we can define, for every real number $\alpha \in \left[0, 1\right]$, the following conditional fuzzy P-measure:

$${\mu }_{\alpha }(g_j/f_i)=\alpha {\mu }_1(g_j/f_i)+(1-\alpha ) {\mu }_2(g_j/f_i).$$

Then we have:

$$\begin{gathered} {\mu }_{\alpha }(f_i\cap g_j)=\alpha {\mu }_1(f_i\cap g_j)+(1-\alpha ) {\mu }_2(f_i\cap g_j), \mathrm{and} \\ {\mu }_{\alpha }(g_j)=\alpha {\mu }_1(g_j)+(1-\alpha ) {\mu }_2(g_j). \end{gathered}$$

Therefore, if we put ${\nu }_{\alpha }(f_i\cap g_j)=\mu (f_i)\cdot {\mu }_{\alpha }(g_j)$, we obtain:

$${\nu }_{\alpha }(f_i\cap g_j)=\alpha \mu (f_i)\cdot {\mu }_1(g_j)+(1-\alpha )\mu (f_i)\cdot {\mu }_2(g_j).$$

According to Theorem 1 and Theorem 10 we can write:

$$\begin{gathered} I_{{\mu }_{\alpha }}(\xi ,\eta )={\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m{\mu }_{\alpha }(f_i\cap g_j)\log \frac{{\mu }_{\alpha }(f_i\cap g_j)}{\mu (f_i)\cdot {\mu }_{\alpha }(g_j)}}}=D_{\eta /\xi }(\mu \Vert {\nu }_{\alpha }) \\ \le \alpha {\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m{\mu }_1(f_i\cap g_j)\log \frac{{\mu }_1(f_i\cap g_j)}{\mu (f_i)\cdot {\mu }_1(g_j)}}}+(1-\alpha ){\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m{\mu }_2(f_i\cap g_j)\log \frac{{\mu }_2(f_i\cap g_j)}{\mu (f_i)\cdot {\mu }_2(g_j)}}} \\ =\alpha I_{{\mu }_1}(\xi ,\eta )+(1-\alpha )I_{{\mu }_2}(\xi ,\eta ). \square \end{gathered}$$

Finally, we define a conditional version of the Kullback-Leibler divergence and, subsequently, we will prove the chain rule for K-L divergence.

Definition 7.

Let $\xi =\left\{f_1,\dots ,f_n\right\},$ $\eta =\left\{g_1,\dots ,g_m\right\}$ be two fuzzy partitions of fuzzy probability spaces $(\Omega , M,\mu ),$ $(\Omega , M,\nu ).$ Then we define the conditional Kullback-Leibler divergence $D_{\eta /\xi }(\mu$ ‖ $\nu )$ by:

$$D_{\eta /\xi }(\mu \Vert \nu )={\displaystyle {\sum }_{i=1}^n\mu (f_i)}{\displaystyle {\sum }_{j=1}^m\dot{\mu }(g_j/f_i)} \log \frac{\dot{\mu }(g_j/f_i)}{\dot{\nu }(g_j/f_i)}.$$

Theorem 12 (Chain rule for K-L divergence).

If $\xi , \eta$ are two fuzzy partitions of fuzzy probability spaces $(\Omega , M,\mu ),$ $(\Omega , M,\nu ),$ then:

$$D_{\xi \vee \eta }(\mu \Vert \nu )=D_{\xi }(\mu \Vert \nu )+ D_{\eta /\xi }(\mu \Vert \nu ).$$

Proof. 

Let $\xi =\left\{f_1,\dots ,f_n\right\},$ $\eta =\left\{g_1,\dots ,g_m\right\}$. Using the property (2.1), we obtain:

$${\displaystyle {\sum }_{j=1}^m\mu (f_i\cap g_j)}=\mu \left(f_i\cap ({\cup }_{j=1}^m{g}_j)\right)=\mu (f_i), \mathrm{for} i=1,2,\dots ,n.$$

Therefore:

$$\begin{gathered} D_{\xi \vee \eta }(\mu \Vert \nu )={\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m\mu (f_i\cap g_j)}} \log \frac{\mu (f_i\cap g_j)}{\nu (f_i\cap g_j)} \\ ={\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m\mu (f_i\cap g_j)}} \log \frac{\mu (f_i) \dot{\mu }(g_j/f_i)}{\nu (f_i) \dot{\nu }(g_j/f_i)} \\ ={\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m\mu (f_i\cap g_j)}} \log \frac{\mu (f_i) }{\nu (f_i) }+{\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^m\mu (f_i\cap g_j)}} \log \frac{ \dot{\mu }(g_j/f_i)}{ \dot{\nu }(g_j/f_i)} \\ ={\displaystyle {\sum }_{i=1}^n\mu (f_i)} \log \frac{\mu (f_i)}{\nu (f_i)}+{\displaystyle {\sum }_{i=1}^n\mu (f_i)}{\displaystyle {\sum }_{j=1}^m\dot{\mu }(g_j/f_i)} \log \frac{\dot{\mu }(g_j/f_i)}{\dot{\nu }(g_j/f_i)} \\ =D_{\xi }(\mu \Vert \nu )+ D_{\eta /\xi }(\mu \Vert \nu ). \square \end{gathered}$$

5. Discussion

In this paper we extend our study concerning mutual information of fuzzy partitions in fuzzy probability spaces. In Section 3, using our previous results, the notion of conditional mutual information of fuzzy partitions is defined and chain rules for mutual information of fuzzy partitions are established. Subsequently, using the notion of conditional mutual information of fuzzy partitions we have defined the notion of conditional independence of fuzzy partitions and we have proved, inter alia, data processing inequality for the studied situation. In Section 4 the notion of Kullback-Leibler divergence for fuzzy P-measures is introduced and the basic properties of this measure are shown. In particular, a convexity of Kullback-Leibler divergence with respect to fuzzy P-measures is proved and a convexity of mutual information of fuzzy partitions is studied. Finally, a conditional version of the Kullback-Leibler divergence is defined and chain rules for Kullback-Leibler divergence with respect to fuzzy P-measures are established.

As noted previously, logical versions of some results concerning the entropy and mutual information of fuzzy partitions presented in Section 2 and Section 3 are given in [10]. In [9] Ellerman studied, in addition to the logical entropy and logical mutual information, the concept of logical Kullback-Leibler divergence. The aim of our next study will be to provide a logical version of Kullback-Leibler divergence for fuzzy probability measures.

Let us mention, finally, the fuzzy set theory has been shown to be useful in many applications of mathematics and it is continually developing. Currently, algebraic structures, based on fuzzy set theory, as MV-algebras (cf. [37,38]), D-posets (cf. [39,40]), effect algebras (cf. [41,42]), and IF-sets (cf. [43,44,45,46,47]) are subject of intensive research and, of course, there are also many results about the entropy on these structures. Some of them can be found, e.g., in [48,49,50,51,52,53,54,55,56,57,58].