# The Kullback–Leibler Information Function for Infinite Measures

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. The Kullback Action for Finite Measures

**Theorem 1**(the local large deviation principle for finite measures)

**.**

**Remark**

**1.**

**Remark**

**2.**

**Remark**

**3.**

## 3. The Kullback Action for σ-Finite Measures

**Example**

**.**Let X be a countable set supplied with the discrete σ-field and μ be the counting measure on X (such that $\mu (x)=1$ for every $x\in X$). Consider a topology on the space of probability distributions ${M}_{1}(X)$ generated by the neighborhoods

**Theorem**

**2.**

**Theorem 3**(the local large deviation principle for infinite measures)

**.**

**Theorem**

**4.**

- (a)
- there exists a weak neighborhood $O({\nu}_{Y})\subset {M}_{1}(Y)$ satisfying the estimate$${\mu}^{n}\left(\right)open="\{"\; close="\}">x=({x}_{1},\cdots ,{x}_{n})\in {Y}^{n}|{\delta}_{x,n}\in O({\nu}_{Y})$$
- (b)
- for any fine neighborhood $O({\nu}_{Y})\subset {M}_{1}(Y)$ and all large enough n,$${\mu}^{n}\left(\right)open="\{"\; close="\}">x=({x}_{1},\cdots ,{x}_{n})\in {Y}^{n}|{\delta}_{x,n}\in O({\nu}_{Y})$$

**Theorem**

**5.**

## 4. Proof of Theorem 2

## 5. Proof of the First Part of Theorem 3

## 6. Proof of the Second Part of Theorem 3

## 7. Proof of Theorems 4 and 5

**Proof**

**of Theorem 4.**

- (a)
- $|\rho ({\nu}_{Y},\mu )-\rho (\nu ,\mu )|<\epsilon $ in the case of finite $\rho (\nu ,\mu )$,
- (b)
- $\rho ({\nu}_{Y},\mu )>1/\epsilon $ in the case $\rho (\nu ,\mu )=+\infty $,
- (c)
- $\rho ({\nu}_{Y},\mu )<-1/\epsilon $ in the case $\rho (\nu ,\mu )=-\infty $.

**Proof**

**of Theorem 5.**

## 8. The Case of Finitely Additive Probability Distributions ν

**Theorem**

**6.**

**Theorem**

**7.**

**Theorem**

**8.**

## 9. Proof of Theorem 6

**Lemma**

**9.**

**Proof.**

**Proof**

**of Theorem 6.**

- (a)
- there exists a positive ε, such that, for any $\delta >0$, one can choose a measurable set ${A}_{\delta}$ such that $\mu ({A}_{\delta})<\delta $ and $\nu ({A}_{\delta})\ge \epsilon $;
- (b)
- the measure ν is improper with respect to μ.

## 10. Proof of Theorems 7 and 8

**Lemma**

**10.**

- (a)
- the restriction of ν to $\mathfrak{B}$ is countably additive;
- (b)
- there exists a fine neighborhood ${O}^{\prime}(\nu )\subset O(\nu )$ generated by $\mathfrak{B}$-measurable functions;
- (c)
- if the measure ν is proper with respect to $\mu \in {M}_{\sigma}(X)$, then the σ-field $\mathfrak{B}$ mentioned above can be chosen in such a way that each of its atoms has a finite measure μ.

**Proof.**

**Proof**

**of Theorem 8.**

## Author Contributions

## Conflicts of Interest

## References

- Billingsley, P. Ergodic Theory and Information; Wiley: New York, NY, USA, 1965; 204p. [Google Scholar]
- Algoet, P.H.; Cover, T.M. A sandwich proof of the Shannon—McMillan—Breiman theorem. Ann. Probab.
**1988**, 16, 899–909. [Google Scholar] [CrossRef] - Sanov, I.N. On the probability of large deviations of random variables. Matematicheskii Sbornik
**1957**, 42, 11–44. (In Russian) [Google Scholar] - Groeneboom, P.; Oosterhoff, J.; Ruymgaart, F.H. Large deviation theorems for empirical probability measures. Ann. Probab.
**1979**, 7, 553–586. [Google Scholar] [CrossRef] - Jain, N. An introduction to large deviations. Lect. Notes Math.
**1985**, 1153, 273–296. [Google Scholar] - Deuschel, J.-D.; Stroockn, D.W. Large Deviations: Pure and Applied Mathematics; Academic Press: Boston, MA, USA, 1989; Volume 137. [Google Scholar]
- Borovkov, A.A.; Mogul’skii, A.A. On large deviation principles in metric spaces. Sib. Math. J.
**2010**, 51, 989–1003. [Google Scholar] [CrossRef] - Acosta, A. On large deviations of empirical measures in the τ-topology. Stud. Appl. Probab.
**1994**, 31A, 41–47. [Google Scholar] [CrossRef] - Bakhtin, V.I. Spectral potential, Kullback action, and large deviations of empirical measures on measurable spaces. Theory Probab. Appl.
**2015**, 59, 535–544. [Google Scholar] [CrossRef] - Bakhtin, V.I. Spectral potential, Kullback action, and large deviation principle for finitely-additive measures. Tr. Inst. Mat.
**2015**, 23, 11–23. (In Russian) [Google Scholar] - Donsker, M.D.; Varadhan, S.R.S. Asymptotic evaluation of certain Markov process expectations for large time, I. Commun. Pure Appl. Math.
**1975**, 28, 1–47. [Google Scholar] [CrossRef] - Donsker, M.D.; Varadhan, S.R.S. Asymptotic evaluation of certain Markov process expectations for large time, II. Commun. Pure Appl. Math.
**1975**, 28, 279–301. [Google Scholar] [CrossRef] - Donsker, M.D.; Varadhan, S.R.S. Asymptotic evaluation of certain Markov process expectations for large time, III. Commun. Pure Appl. Math.
**1976**, 29, 389–461. [Google Scholar] [CrossRef] - Donsker, M.D.; Varadhan, S.R.S. Asymptotic evaluation of certain Markov process expectations for large time, IV. Commun. Pure Appl. Math.
**1983**, 36, 183–212. [Google Scholar] [CrossRef] - Varadhan, S.R.S. Large deviations. Ann. Probab.
**2008**, 36, 397–419. [Google Scholar] [CrossRef] - Freidlin, M.I.; Wentzell, A.D. Random Perturbations of Dynamical Systems, 3rd ed.; Springer: Berlin, Germany, 2012; 458p. [Google Scholar]
- Borovkov, A.A.; Mogul’skii, A.A. Large deviation principles for random walk trajectories. I. Theory Probab. Appl.
**2012**, 56, 538–561. [Google Scholar] [CrossRef] - Borovkov, A.A.; Mogul’skii, A.A. Large deviation principles for random walk trajectories. II. Theory Probab. Appl.
**2013**, 57, 1–27. [Google Scholar] [CrossRef] - Borovkov, A.A.; Mogul’skii, A.A. Large deviation principles for random walk trajectories. III. Theory Probab. Appl.
**2014**, 58, 25–37. [Google Scholar] [CrossRef] - Borovkov, A.A. Mathematical Statistics; Gordon & Breach: Amsterdam, The Netherlands, 1998; 570p. [Google Scholar]
- Sokol, E.E. A generalization of McMillan’s theorem on the case of countable alphabet. Tr. Inst. Mat.
**2015**, 23, 115–122. (In Russian) [Google Scholar] - Sokal, E.E. On the informational sense of entropy in the case of countable alphabet. Vestn. Beloruss. Gos. Univ. Ser. 1 Fiz. Mat. Inform.
**2016**, 1, 96–100. (In Russian) [Google Scholar]

© 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Bakhtin, V.; Sokal, E.
The Kullback–Leibler Information Function for Infinite Measures. *Entropy* **2016**, *18*, 448.
https://doi.org/10.3390/e18120448

**AMA Style**

Bakhtin V, Sokal E.
The Kullback–Leibler Information Function for Infinite Measures. *Entropy*. 2016; 18(12):448.
https://doi.org/10.3390/e18120448

**Chicago/Turabian Style**

Bakhtin, Victor, and Edvard Sokal.
2016. "The Kullback–Leibler Information Function for Infinite Measures" *Entropy* 18, no. 12: 448.
https://doi.org/10.3390/e18120448