Asymptotic Properties of a Statistical Estimator of the Jeffreys Divergence: The Case of Discrete Distributions

Abstract

We investigate the asymptotic properties of the plug-in estimator for the Jeffreys divergence, the symmetric variant of the Kullback–Leibler (KL) divergence. This study focuses specifically on the divergence between discrete distributions. Traditionally, estimators rely on two independent samples corresponding to two distinct conditions. However, we propose a one-sample estimator where the condition results from a random event. We establish the estimator’s asymptotic unbiasedness (law of large numbers) and asymptotic normality (central limit theorem). Although the results are expected, the proofs require additional technical work due to the randomness of the conditions.

1. Introduction

The Kullback–Leibler (KL) divergence was developed as an extension of Shannon’s information; see [1]. This measure is widely used in various research fields where there is a need to compare two (probability) measures. Significant research has been focused on generalizing this concept. KL divergence is considered nowadays as a special case of f-divergence measures, with $f(·)=ln(·)$. In turn, this class of divergence is a part of a broader class of divergences known as information divergences. For a literature review of various classes of information divergence measures, we refer the reader to [2]. For the estimation of f-divergence, refer to [3] and the references therein.

In mathematical statistics, the Kullback–Leibler (KL) divergence introduced in [1], also called relative entropy in information theory, is a type of statistical measure used to quantify the dissimilarity between two probability measures. This measure of divergence has been widely used in various fields, such as variational inference [4,5], Bayesian inference [6,7], metric learning [8,9,10], machine learning [11,12], computer vision [13,14], physics [15], biology [16], and information geometry [17], among many other application fields. It is worth mentioning works related to the application of various types of divergences for goodness-of-fit problems, where theoretical and empirical distributions are compared. For instance, see [18,19,20] and the references therein.

We consider an estimation problem of the symmetrized version of KL divergence known as Jeffreys divergence. There is a vast body of literature dedicated to the statistical estimation of information-type divergence. Most of the works are dedicated to continuous-type distributions, because, in general, it is hard to calculate the definite integral in the definition of the convergence. See [21,22] and references within. However, here, we focused on discrete (or categorical) random variables.

Let $\mathcal{X}=\{a_1,a_2,\dots ,a_r\}$ be a finite set, where $r\ge 2$. A random variable with values from $\mathcal{X}$ is characterized by the mass distribution $\mathbf{p}={\left(p_a\right)}_{a\in \mathcal{X}}$. Let $\mathcal{P}\left(\mathcal{X}\right)$ be the set of all (positive) distributions on $\mathcal{X}$,

$$\mathcal{P}\left(\mathcal{X}\right)=\left\{\mathbf{p}={\left(p_a\right)}_{a\in \mathcal{X}}\in {\mathbb{R}}^r:p_a>0\mathrm{and}\sum _{a\in \mathcal{X}}{p}_a=1\right\}.$$

For any two $\mathbf{p},\mathbf{q}\in \mathcal{P}\left(\mathcal{X}\right)$, the KL divergence between $\mathbf{p}$ and $\mathbf{q}$ is defined as follows

$${\mathcal{D}}_{KL}\left(\mathbf{p}\right|\left|\mathbf{q}\right)=\sum _{a\in \mathcal{X}}{p}_{a}ln\left({\displaystyle \frac{p_a}{q_a}}\right).$$

KL divergence is asymmetric, and the values of ${\mathcal{D}}_{KL}\left(\mathbf{p}\right|\left|\mathbf{q}\right)$ and ${\mathcal{D}}_{KL}\left(\mathbf{q}\right|\left|\mathbf{p}\right)$ are different in general. This lack of symmetry, in some specific contexts, can be a disadvantage when measuring similarities between probability measures; see, e.g., [17,23,24,25,26,27,28]. Thus, it is often quite useful to work with a symmetrization of the KL divergence which is defined as

$$\begin{array}{ccc}\hfill {\mathcal{D}}_{KL}^{sym}\left(\mathbf{p}\right|\left|\mathbf{q}\right)& :=& {\mathcal{D}}_{KL}\left(\mathbf{p}\right|\left|\mathbf{q}\right)+{\mathcal{D}}_{KL}\left(\mathbf{q}\right|\left|\mathbf{p}\right),\hfill \\ & =& \sum _{a\in \mathcal{X}}(p_a-q_a)ln\left({\displaystyle \frac{p_a}{q_a}}\right).\hfill \end{array}$$

In [1], Kullback and Leibler consider this symmetrized version and refer to it as the divergence between the two measures. This concept was earlier introduced and studied by Harold Jeffreys in 1948, with some papers and Wikipedia citing the second edition of [29]. Today, this symmetrization is known as Jeffreys divergence. For some advantages and applications of including symmetrization in KL divergence, see, for example, [30,31,32,33,34,35]. Additionally, the Jeffreys divergence is related to the Population Stability Index (PSI) used in finance and serves as a foundation of the Cluster Validity Index (CVI), as discussed in [36].

The problem of estimating information divergence (and entropy) for discrete random variables is well established in the literature; see, for example, [37,38,39,40,41,42,43]. It is important to note that these references do not represent a comprehensive list of all relevant works; they are merely examples that illustrate the diversity of approaches in this field. In these works, the authors examine the convergence properties of estimators derived through various methods for discrete distributions. They analyze the asymptotic behavior and provide theoretical insights into the performance of these estimators, contributing to a deeper understanding of various types of divergence and related measures. All such studies known to us generally operate within the following framework. Given two independent samples $X_1^{\left(1\right)},X_2^{\left(1\right)},\dots ,X_l^{\left(1\right)}$ and $X_1^{\left(2\right)},X_2^{\left(2\right)},\dots ,X_m^{\left(2\right)}$ drawn from two discrete distributions $\mathbf{p}$ and $\mathbf{q}$, respectively, the goal is to develop an estimator for the divergence based on the sample sizes l and m and to analyze their convergence.

However, in practice, the following scenario arises. It involves paired sample observations: $(X,Y)$, $(X_1,Y_1)$, $(X_2,Y_2)$, …, $(X_n,Y_n)$, where $Y_i\in \{0,1\}$ indicates a binary condition, and $\mathbf{p}$ and $\mathbf{q}$ represent the conditional distributions $\mathbf{P}(·\mid Y=0)$ and $\mathbf{P}(·\mid Y=1)$, respectively. In other words, the sample sizes l and m mentioned above are realizations of a random experiment and should be treated as random variables, where l has a binomial distribution with success probability $\mathbf{P}(Y=0)$ in a sequence of n trials, and $m=n-l$. Below, in Section 2.1, we consider an example of this scenario.

This paper specifically focuses on this scenario, establishing the asymptotic properties of the plug-in estimator within this framework. To the best of our knowledge, there are few theoretical works on the asymptotic properties of estimating the Jeffreys divergence, and none directly address the framework we consider. While the standard $\delta$-method can be employed to derive the central limit theorem (CLT), it typically yields a general and cumbersome variance formula involving the product of large matrices. In contrast, our direct and probabilistic approach provides a more explicit and straightforward variance formula. An additional benefit of our approach is that the results obtained can be effectively integrated into undergraduate statistics courses.

The paper is organized as follows. In the next section, Section 2, we introduce definitions and formulate the asymptotic results with proofs. Auxiliary results are provided in Section 3.

2. Notations and Main Results

Let $(X,Y)$ be a bivariate random vector with values from $\mathcal{X}\times \{0,1\}$ defined on the probability space $(\Omega ,\mathcal{A},\mathbf{P})$. Recall that $\mathcal{X}=\{a_1,a_2,\dots ,a_r\}$ is a finite set, where $r\ge 2$. We denote by $\mathbf{E}$ the expectation with respect to $\mathbf{P}$. The (marginal) distribution of Y is a Bernoulli distribution with success probability $p:=\mathbf{P}(Y=1)\in (0,1)$, and let $q:=1-p=\mathbf{P}(Y=0)$. Let $\mathbf{p}={\left(p_j\right)}_{1\le j\le r}$ and $\mathbf{q}={\left(q_j\right)}_{1\le j\le r}$ two conditional distributions of X,

$$p_j:=\mathbf{P}(X=a_j\mid Y=1)\mathrm{and}{q}_j:=\mathbf{P}(Y=a_j\mid Y=0),$$

for any $j=1,\dots ,r$. We will assume that all probabilities above are positive,

$$\underset{1\le j\le r}{min}\{p_j,q_j\}>0,$$

and $\mathbf{p}\ne \mathbf{q}$, i.e. there exists at least one j, such that $p_j\ne q_j$. Moreover, we do not assume that the number r is known. Consider the empirical probability measures ${\widehat{\mathbf{p}}}_n={\left({\widehat{p}}_{j,n}\right)}_{1\le j\le r}$ and ${\widehat{\mathbf{q}}}_n={\left({\widehat{q}}_{j,n}\right)}_{1\le j\le r}$, generated by the sequence i.i.d. random variables $(X_1,Y_1)\dots ,(X_n,Y_n)$ from the bivariate distribution, $(\mathbf{p},\mathbf{q})$, defined from

$${\widehat{p}}_{j,n}:={\displaystyle \frac{{\displaystyle {\displaystyle \sum _{i=1}^n}}\mathbf{I}(X_i=a_j,Y_i=1)}{{\displaystyle \sum _{i=1}^n}\mathbf{I}(Y_i=1)}},{\widehat{q}}_{j,n}:={\displaystyle \frac{{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=0)}{{\displaystyle \sum _{i=1}^n}\mathbf{I}(Y_i=0)}},$$

(1)

where $\mathbf{I}\left(A\right)$ is an indicator of an event A. We assume that the above fractions are zero when the corresponding denominator takes the zero value. Additionally, consider the following notations

$${\widehat{p}}_n:={\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbf{I}(Y_i=1),{\widehat{q}}_n:={\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbf{I}(Y_i=0).$$

(2)

Based on (1), we define the plug-in estimator for the Jeffreys divergence ${\mathcal{D}}_{KL}^{sym}\left(\mathbf{p}\right|\left|\mathbf{q}\right)$ as follows,

$${\mathcal{D}}_{KL}^{sym}({\widehat{\mathbf{p}}}_n\left|\right|{\widehat{\mathbf{q}}}_n):=\sum _{j=1}^r({\widehat{p}}_{j,n}-{\widehat{q}}_{j,n})ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}.$$

(3)

where the standard convention is adopted: if $x\ge 0$, then $0log(0/x)=0$, and if $x>0$, then $xlog(x/0)=\infty$. Note that the estimator appears to depend on r. However, for any symbol $a_j\in \mathcal{X}$ with ${\widehat{p}}_{j,n}=0$ or ${\widehat{q}}_{j,n}=0$, the corresponding term is not included in the sum. Thus, the sum contains only the terms corresponding to the observed symbols from $\mathcal{X}$, and we do not need to know r.

In this paper, we study the convergence properties, specifically, the asymptotic unbiasedness, or law of large numbers (LLN), and asymptotic normality, or central limit theorem (CLT), of the plug-in estimator (3). To this end, denote

$$\begin{array}{ccc}\hfill {\eta }_n& :=& {\mathcal{D}}_{KL}^{sym}({\widehat{\mathbf{p}}}_n\left|\right|{\widehat{\mathbf{q}}}_n)-{\mathcal{D}}_{KL}^{sym}\left(\mathbf{p}\right|\left|\mathbf{q}\right)\hfill \\ & =& \sum _{j=1}^r\left(({\widehat{p}}_{j,n}-{\widehat{q}}_{j,n})ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}-(p_j-q_j)ln{\displaystyle \frac{p_j}{q_j}}\right).\hfill \end{array}$$

Theorem 1. 

(Law of Large Numbers) The following equality holds

$$\underset{n\to \infty }{lim}{\eta }_n=0a.s.$$

Proof. 

It follows directly from the strong law of large numbers for sequences ${\widehat{p}}_n$, ${\widehat{q}}_n$, ${\widehat{p}}_{j,n}$, ${\widehat{q}}_{j,n}$, $1\le j\le r$, defined in (1) and (2). □

Theorem 2. 

(Central Limit Theorem) The following convergence holds true

$$\sqrt{n}{\eta }_n\stackrel{d}{\to }\xi \sim N(0,{\sigma }^2),asn\to \infty ,$$

where $\stackrel{d}{\to }$ means the convergence in distribution,

$$\begin{array}{cc}\hfill {\sigma }^2=\mathbf{E}& \left(\sum _{j=1}^r\right(\left({\displaystyle \frac{1}{p}}(\mathbf{I}(X=a_j,Y=1)-p{p}_j)-p_j(\mathbf{I}(Y=1)-p)\right)\left(1+ln{\displaystyle \frac{p_j}{q_j}}-{\displaystyle \frac{q_j}{p_j}}\right)\hfill \\ & +\left({\displaystyle \frac{1}{q}}(\mathbf{I}(X=a_j,Y=0)-q{q}_j)-q_j(\mathbf{I}(Y=0)-q)\right)\left(1+ln{\displaystyle \frac{q_j}{p_j}}-{\displaystyle \frac{p_j}{q_j}}\right)){)}^2.\hfill \end{array}$$

The plug-in estimator of variance yields the following corollary.

Corollary 1. 

The following convergence holds true

$$\sqrt{n}{\displaystyle \frac{{\eta }_n}{{\widehat{\sigma }}_n}}\stackrel{d}{\to }\tilde{\xi }\sim N(0,1),asn\to \infty ,$$

where

$$\begin{array}{cc}\hfill {\widehat{\sigma }}_n^2=& {\displaystyle \frac{1}{n}}\sum _{i=1}^n\left(\sum _{j=1}^r\right[\left({\displaystyle \frac{1}{{\widehat{p}}_n}}(\mathbf{I}(X_i=a_j,Y_i=1)-{\widehat{p}}_n{\widehat{p}}_{j,n})-{\widehat{p}}_{j,n}(\mathbf{I}(Y_i=1)-{\widehat{p}}_n)\right)\left(1+ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}-{\displaystyle \frac{{\widehat{q}}_{j,n}}{{\widehat{p}}_{j,n}}}\right)\hfill \\ & +\left({\displaystyle \frac{1}{{\widehat{q}}_n}}(\mathbf{I}(X_i=a_j,Y_i=0)-{\widehat{q}}_n{\widehat{q}}_{j,n})-{\widehat{q}}_{j,n}(\mathbf{I}(Y_i=0)-{\widehat{q}}_n)\right)\left(1+ln{\displaystyle \frac{{\widehat{q}}_{j,n}}{{\widehat{p}}_{j,n}}}-{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}\right)]{)}^2.\hfill \end{array}$$

Indeed, the sequence ${\widehat{\sigma }}_n^2$, by the law of large numbers, converges in probability to ${\sigma }^2$, and hence the proof of the corollary directly follows from Theorem 2 and Slutsky’s theorem. Recall that Slutsky’s theorem states that if a sequence of random variables converges in distribution and another sequence converges in probability to a constant, then their product (or sum) also converges in distribution, which is applicable in this context to complete the proof of this corollary.

Let us now proceed with the proof of Theorem 2.

Proof. 

To prove the theorem, we will show that the asymptotics of the sequence of interest matches those of the sequence in Lemma 3, involving the differences ${\widehat{p}}_{j,n}-p_j$ and ${\widehat{q}}_{j,n}-q_j$ for $j=1,\dots ,r$. To achieve this, we need some technical work, isolating the relevant term from Lemma 3 and the term that converges to zero in probability. We rewrite the $\sqrt{n}{\eta }_n$ in the following way:

$$\begin{array}{cc}\hfill \sqrt{n}{\eta }_n=& \sqrt{n}\sum _{j=1}^r\left(({\widehat{p}}_{j,n}-{\widehat{q}}_{j,n})ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}-(p_j-q_j)ln{\displaystyle \frac{p_j}{q_j}}\right)\hfill \\ \hfill =& \sqrt{n}\sum _{j=1}^r\left(({\widehat{p}}_{j,n}-{\widehat{q}}_{j,n})ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}\pm (p_j-q_j)ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}-(p_j-q_j)ln{\displaystyle \frac{p_j}{q_j}}\right)\hfill \\ \hfill =& \sqrt{n}\sum _{j=1}^r(({\widehat{p}}_{j,n}-p_j)-({\widehat{q}}_{j,n}-q_j))ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}+\sqrt{n}\sum _{j=1}^r\left(p_j-q_j\right)ln{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}\hfill \\ \hfill =& \sqrt{n}\sum _{j=1}^r(({\widehat{p}}_{j,n}-p_j)-({\widehat{q}}_{j,n}-q_j))ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}+\sqrt{n}\sum _{j=1}^r\left(p_j-q_j\right)ln\left(1+{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right).\hfill \end{array}$$

(4)

Denote

$$y_{j,n}:=ln\left(1+{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right).$$

We will use the Taylor expansion for a function $f\left(x\right)$ with a remainder term in Lagrange form (see e.g., [44] (p. 880)): for any x with $\left|x\right|<1$, there exists a positive $K_x\in (0,1)$ such that

$$f\left(x\right)=f\left(0\right)+{\displaystyle \frac{f^{\prime }\left(0\right)}{1!}}x+{\displaystyle \frac{f^{\prime \prime }\left(K_{x}x\right)}{2!}}{x}^2.$$

Applying this to the function $f\left(x\right)=ln(1+x)$, we have

$$ln(1+x)=x-{\displaystyle \frac{x^2}{2{(1+K_{x}x)}^2}},$$

(5)

for $\left|x\right|<1$, where $K_x\in (0,1)$. It is easy to see that the following upper bound holds when $\left|x\right|\le {\displaystyle \frac{1}{2}}$ and $K_x\in (0,1)$:

$${\displaystyle \frac{x^2}{2{(1+K_{x}x)}^2}}\le 2x^2.$$

(6)

Applying (5) and (6) with $x={\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}$, we obtain

$$\begin{array}{cc}\hfill \left|y_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|\le & 2{\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right)}^2\mathbf{I}\left(\left|{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|\le {\displaystyle \frac{1}{2}}\right)\hfill \\ & +\left|y_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|\mathbf{I}\left(\left|{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|>{\displaystyle \frac{1}{2}}\right)\mathrm{a}.\mathrm{s}.\hfill \end{array}$$

(7)

for any $1\le j\le r$. Using (7) and upper bounds (17) and (18) from Lemma 1 in Section 3 (Auxiliary Results), we obtain the following: for any $\epsilon >0$, $1\le j\le r$ and for some constant $C>0$,

$$\begin{array}{cc}\hfill \mathbf{P}& \left(\sqrt{n}\left|y_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|>\epsilon \right)\hfill \\ \hfill \le & \mathbf{P}\left(2{\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right)}^2\mathbf{I}\left(\left|{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|\le {\displaystyle \frac{1}{2}}\right)>{\displaystyle \frac{\epsilon }{2\sqrt{n}}}\right)\hfill \\ & +\mathbf{P}\left(\sqrt{n}\left|y_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|\mathbf{I}\left(\left|{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|>{\displaystyle \frac{1}{2}}\right)>{\displaystyle \frac{\epsilon }{2}}\right)\hfill \\ \hfill \le & \mathbf{P}\left({\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right)}^2>{\displaystyle \frac{\epsilon }{4\sqrt{n}}}\right)+\mathbf{P}\left(\left|{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|>{\displaystyle \frac{1}{2}}\right)\hfill \\ \hfill \le & 2\mathbf{P}\left(\left|{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}\right|>{\displaystyle \frac{min(\sqrt{\epsilon },1)}{2n^{\frac{1}{4}}}}\right)\hfill \\ \hfill \le & 2\mathbf{P}\left(q_j\left|{\widehat{p}}_{j,n}-p_j\right|+p_j\left|{\widehat{q}}_{j,n}-q_j\right|>{\displaystyle \frac{min(\sqrt{\epsilon },1)p_j{q}_j}{4n^{\frac{1}{4}}}},{\widehat{q}}_{j,n}\ge {\displaystyle \frac{q_j}{2}}\right)+2\mathbf{P}\left({\widehat{q}}_{j,n}<{\displaystyle \frac{q_j}{2}}\right)\hfill \\ \hfill \le & 2\mathbf{P}\left(\left|{\widehat{p}}_{j,n}-p_j\right|>{\displaystyle \frac{min(\sqrt{\epsilon },1)p_j}{8n^{\frac{1}{4}}}}\right)+2\mathbf{P}\left(\left|{\widehat{q}}_{j,n}-q_j\right|>{\displaystyle \frac{min(\sqrt{\epsilon },1)q_j}{8n^{\frac{1}{4}}}}\right)\hfill \\ & +2\mathbf{P}\left(|q_j-{\widehat{q}}_{j,n}|>{\displaystyle \frac{q_j}{2}}\right)\le 4exp\{-\sqrt{n}C\}+2exp\{-nC\}.\hfill \end{array}$$

(8)

Denote

$$z_{j,n}:={\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}},v_{j,n}:={\displaystyle \frac{{\widehat{q}}_{j,n}-q_j}{q_j}}.$$

For any $1\le j\le r$, we have

$$\begin{array}{cc}\hfill \left|z_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{q}_j}}\right|=& {\displaystyle \frac{1}{p_j{q}_j}}\left|{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{1+\frac{{\widehat{q}}_{j,n}-q_j}{q_j}}}-({\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n})\right|\hfill \\ \hfill =& {\displaystyle \frac{1}{p_j{q}_j}}\left|({\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n})\left(-v_{j,n}+\sum _{k=2}^{\infty }{\left(-v_{j,n}\right)}^k\right)\right|\mathbf{I}\left(\left|v_{j,n}\right|\le {\displaystyle \frac{1}{2}}\right)\hfill \\ & +\left|z_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{q}_j}}\right|\mathbf{I}\left(\left|v_{j,n}\right|>{\displaystyle \frac{1}{2}}\right)\hfill \\ \hfill \le & {\displaystyle \frac{2}{p_j}}|{\widehat{p}}_{j,n}-p_j|·|v_{j,n}|\mathbf{I}\left(\left|v_{j,n}\right|\le {\displaystyle \frac{1}{2}}\right)+{\displaystyle \frac{2}{q_j}}|{\widehat{q}}_{j,n}-q_j|·|v_{j,n}|\mathbf{I}\left(\left|v_{j,n}\right|\le {\displaystyle \frac{1}{2}}\right)\hfill \\ & +\left|z_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{q}_j}}\right|\mathbf{I}\left(\left|v_{j,n}\right|>{\displaystyle \frac{1}{2}}\right)\mathrm{a}.\mathrm{s}.\hfill \end{array}$$

(9)

Using the last bound (9) together with upper bounds (17) and (18) from Lemma 1, we obtain the following: for any $\epsilon >0$, $1\le j\le r$ and for some constant $C>0$,

$$\begin{array}{cc}\hfill \mathbf{P}& \left(\sqrt{n}\left|z_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{q}_j}}\right|>\epsilon \right)\hfill \\ \hfill \le & \mathbf{P}\left(\sqrt{n}\left|z_{j,n}-{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{q}_j}}\right|\mathbf{I}\left(\left|v_{j,n}\right|>{\displaystyle \frac{1}{2}}\right)>{\displaystyle \frac{\epsilon }{3}}\right)\hfill \\ & +\mathbf{P}\left(|{\widehat{p}}_{j,n}-p_j|·|v_{j,n}|\mathbf{I}\left(\left|v_{j,n}\right|\le {\displaystyle \frac{1}{2}}\right)>{\displaystyle \frac{\epsilon p_j}{6\sqrt{n}}}\right)\hfill \\ & +\mathbf{P}\left(|{\widehat{q}}_{j,n}-q_j|·|v_{j,n}|\mathbf{I}\left(\left|v_{j,n}\right|\le {\displaystyle \frac{1}{2}}\right)>{\displaystyle \frac{\epsilon q_j}{6\sqrt{n}}}\right)\hfill \\ \hfill \le & \mathbf{P}\left(|{\widehat{p}}_{j,n}-p_j|>{\displaystyle \frac{\sqrt{\epsilon p_j}}{\sqrt{6}{n}^{\frac{1}{4}}}}\right)+\mathbf{P}\left(|v_{j,n}|>{\displaystyle \frac{\sqrt{\epsilon p_j}}{\sqrt{6}{n}^{\frac{1}{4}}}}\right)\hfill \\ & +\mathbf{P}\left(|{\widehat{q}}_{j,n}-q_j|>{\displaystyle \frac{\sqrt{\epsilon q_j}}{\sqrt{6}{n}^{\frac{1}{4}}}}\right)+\mathbf{P}\left(|v_{j,n}|>{\displaystyle \frac{\sqrt{\epsilon q_j}}{\sqrt{6}{n}^{\frac{1}{4}}}}\right)+\mathbf{P}\left(\left|v_{j,n}\right|>{\displaystyle \frac{1}{2}}\right)\hfill \\ \hfill \le & 4exp\{-C\sqrt{n}\}+exp\{-Cn\}.\hfill \end{array}$$

(10)

Once more, the inequalities (17) and (18) from Lemma 1 and Lemma 2 (see Section 3) give us the following: for any $\epsilon >0$, $1\le j\le r$ and some constant $C>0$,

$$\begin{array}{cc}\hfill \mathbf{P}& \left(\sqrt{n}\left|(({\widehat{p}}_{j,n}-p_j)-({\widehat{q}}_{j,n}-q_j))ln{\displaystyle \frac{{\widehat{p}}_{j,n}}{{\widehat{q}}_{j,n}}}-(({\widehat{p}}_{j,n}-p_j)-({\widehat{q}}_{j,n}-q_j))ln{\displaystyle \frac{p_j}{q_j}}\right|\ge \epsilon \right)\hfill \\ & \le \mathbf{P}\left(|{\widehat{p}}_{j,n}-p_j|·\left|ln{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}\right|\ge {\displaystyle \frac{\epsilon }{2\sqrt{n}}}\right)+\mathbf{P}\left(|{\widehat{q}}_{j,n}-q_j|·\left|ln{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}\right|\ge {\displaystyle \frac{\epsilon }{2\sqrt{n}}}\right)\hfill \\ & \le \mathbf{P}\left(|{\widehat{p}}_{j,n}-p_j|\ge {\displaystyle \frac{\sqrt{\epsilon }}{\sqrt{2}{n}^{\frac{1}{4}}}}\right)+2\mathbf{P}\left(\left|ln{\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}\right|\ge {\displaystyle \frac{\sqrt{\epsilon }}{\sqrt{2}{n}^{\frac{1}{4}}}}\right)+\mathbf{P}\left(|{\widehat{q}}_{j,n}-q_j|\ge {\displaystyle \frac{\sqrt{\epsilon }}{\sqrt{2}{n}^{\frac{1}{4}}}}\right)\hfill \\ & \le 4exp\{-C\sqrt{n}\}.\hfill \end{array}$$

(11)

Denote

$${\xi }_n:=\sqrt{n}\sum _{j=1}^r\left(({\widehat{p}}_{j,n}-p_j)\left(1+ln{\displaystyle \frac{p_j}{q_j}}-{\displaystyle \frac{q_j}{p_j}}\right)+({\widehat{q}}_{j,n}-q_j))\left(1+ln{\displaystyle \frac{q_j}{p_j}}-{\displaystyle \frac{p_j}{q_j}}\right)\right).$$

Relations (4), (8), (10), and (11) imply that for any $\epsilon >0$ there exist constants $C_1>0$ and $C>0$ such that

$$\mathbf{P}\left(\right|\sqrt{n}{\eta }_n-{\xi }_n|>\epsilon )\le C_{1}exp\{-C\sqrt{n}\}.$$

Thus, thanks to Slutsky’s theorem, the random sequences ${\eta }_n$ and ${\xi }_n$ have the same limit by distribution. Finally, Lemma 3 with

$$b_j=1+ln{\displaystyle \frac{p_j}{q_j}}-{\displaystyle \frac{q_j}{p_j}},c_j=1+ln{\displaystyle \frac{q_j}{p_j}}-{\displaystyle \frac{p_j}{q_j}},$$

concludes the proof. □

2.1. Example

We show how to cluster catastrophic processes (see, [45,46]), using Jeffreys Divergence for their characteristics. Suppose that we have two types of insurance claims. Let $Y\in \{0,1\}$ be a type of claim with $\mathbf{P}(Y=1)=p$ and $\mathbf{P}(Y=0)=q$, and let X be the size of damage, or payment according to the claim. The conditional distribution of X is $\mathbf{P}(X=a_j|Y=1)=p_j$, $\mathbf{P}(X=a_j|Y=0)=q_j$, where $a_j$, $1\le j\le r$. We assume that there is no difference between the conditional distribution of X if

$${\mathcal{D}}_{KL}^{sym}\left(\mathbf{p}\right|\left|\mathbf{q}\right)=\sum _{j=1}^r(p_j-q_j)ln\left({\displaystyle \frac{p_j}{q_j}}\right)<\alpha ,$$

(12)

where $\alpha$ is a small critical value that distinguishes the types of claims. Let $(X_1,Y_1),\dots ,(X_n,Y_n)$ be the sample of size n, and let ${\mathcal{D}}_{KL}^{sym}({\widehat{\mathbf{p}}}_n\left|\right|{\widehat{\mathbf{q}}}_n)$ and ${\widehat{\sigma }}_n$ be the respective estimations. For sufficiently large n, we can estimate the probability (see Corollary 1)

$$\begin{array}{cc}\hfill \mathbf{P}\left({\mathcal{D}}_{KL}^{sym}<\alpha \right)& =\mathbf{P}\left({\eta }_n>{\mathcal{D}}_{KL}^{sym}({\widehat{\mathbf{p}}}_n\left|\right|{\widehat{\mathbf{q}}}_n)-\alpha \right)=\mathbf{P}\left({\eta }_n>{\mathcal{D}}_{KL}^{sym}({\widehat{\mathbf{p}}}_n\left|\right|{\widehat{\mathbf{q}}}_n)-\alpha \right)\hfill \\ & =\mathbf{P}\left(\sqrt{n}{\displaystyle \frac{{\eta }_n}{{\widehat{\sigma }}_n}}>\sqrt{n}{\displaystyle \frac{{\mathcal{D}}_{KL}^{sym}({\widehat{\mathbf{p}}}_n\left|\right|{\widehat{\mathbf{q}}}_n)-\alpha }{{\widehat{\sigma }}_n}}\right)\approx {\displaystyle \frac{1}{\sqrt{2\pi }}}{\int }_{b_n}^{\infty }{e}^{-{\displaystyle \frac{t^2}{2}}}dt,\hfill \end{array}$$

where

$$b_n=\sqrt{n}{\displaystyle \frac{{\mathcal{D}}_{KL}^{sym}({\widehat{\mathbf{p}}}_n\left|\right|{\widehat{\mathbf{q}}}_n)-\alpha }{{\widehat{\sigma }}_n}}.$$

These relations propose a calculation of the probability that inequality (12) is true.

3. Auxiliary Results

The following lemma contains the proofs of inequalities (17) and (18) used in the proof of Theorem 2. The other inequalities of the lemma, (13)–(16), are used in the proofs in this section.

Lemma 1. 

For any $g>0$, the following inequalities hold

$$\mathbf{P}\left(\left|{\widehat{p}}_n-p)\right|>g\right)\le 2exp\left\{-{\displaystyle \frac{n{g}^2}{2{(max(p,q))}^2}}\right\},$$

(13)

$$\mathbf{P}\left(\left|{\widehat{q}}_n-q)\right|>g\right)\le 2exp\left\{-{\displaystyle \frac{n{g}^2}{2{(max(p,q))}^2}}\right\},$$

(14)

$$\underset{1\le j\le r}{max}\mathbf{P}\left({\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(X_i=a_j,Y_i=1)-p{p}_j)>g\right)\le 2exp\left\{-{\displaystyle \frac{n{g}^2}{2{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\},$$

(15)

$$\underset{1\le j\le r}{max}\mathbf{P}\left({\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(X_i=a_j,Y_i=0)-q{q}_j)>g\right)\le 2exp\left\{-{\displaystyle \frac{n{g}^2}{2{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\},$$

(16)

$$\begin{array}{cc}\hfill \underset{1\le j\le r}{max}& \mathbf{P}\left(\left|{\widehat{p}}_{j,n}-p_j\right|>g\right)\hfill \\ \hfill \le & 2exp\left\{-{\displaystyle \frac{n{g}^2{p}^2}{2^7{(max(p,q))}^2{p}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{g}^2{p}^2}{8{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}\hfill \\ & +exp\left\{-{\displaystyle \frac{n{p}^2{p}_{min}^2}{2{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}+exp\left\{-{\displaystyle \frac{n{p}^2}{8{(max(p,q))}^2}}\right\},\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill \underset{1\le j\le r}{max}& \mathbf{P}\left(\left|{\widehat{q}}_{j,n}-q_j\right|>g\right)\hfill \\ \hfill \le & 2exp\left\{-{\displaystyle \frac{n{g}^2{q}^2}{2^7{(max(p,q))}^2{q}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{g}^2{q}^2}{8{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}\hfill \\ & +exp\left\{-{\displaystyle \frac{n{q}^2{q}_{min}^2}{2{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}+exp\left\{-{\displaystyle \frac{n{q}^2}{8{(max(p,q))}^2}}\right\},\hfill \end{array}$$

(18)

where $p_{min}:=\underset{1\le j\le r}{min}{p}_j$, $p_{max}:=\underset{1\le j\le r}{max}{p}_j$, $q_{min}:=\underset{1\le j\le r}{min}{p}_j$, $q_{max}:=\underset{1\le j\le r}{max}{p}_j$.

Proof. 

The Hoeffding inequality proves (13). Indeed,

$$\begin{array}{cc}\hfill \mathbf{P}\left(\left|{\widehat{p}}_n-p)\right|>g\right)=& \mathbf{P}\left(\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|>g\right)\hfill \\ \hfill \le & \mathbf{P}\left(\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)>ng\right)+\mathbf{P}\left(-\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)>ng\right)\hfill \\ \hfill \le & 2exp\left\{-{\displaystyle \frac{n{g}^2}{2{(max(p,q))}^2}}\right\}.\hfill \end{array}$$

Inequalities (14)–(16) are obtained in the same way, and therefore we omit their proofs.

Let us prove (17). We have

$$\begin{array}{cc}\hfill \mathbf{P}\left(\left|{\widehat{p}}_{j,n}-p_j\right|>g\right)=& \mathbf{P}\left(\left|{\displaystyle \frac{\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=1)}{\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(Y_i=1)}}\pm {\displaystyle \frac{\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=1)}{p}}-p_j\right|>g\right)\hfill \\ \hfill \le & \mathbf{P}\left(\left|{\displaystyle \frac{\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=1)}{\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(Y_i=1)}}-{\displaystyle \frac{\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=1)}{p}}\right|>{\displaystyle \frac{g}{2}}\right)\hfill \\ & +\mathbf{P}\left(\left|{\displaystyle \frac{\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=1)}{p}}-p_j\right|>{\displaystyle \frac{g}{2}}\right)=:{\mathbf{P}}_1+{\mathbf{P}}_2.\hfill \end{array}$$

(19)

Denote

$$A:=\left\{{\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)>2p{p}_j\right\},B:=\left\{{\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbf{I}(Y_i=1)<{\displaystyle \frac{p}{2}}\right\}.$$

Let us find the upper bound for the probability ${\mathbf{P}}_1$. We have

$$\begin{array}{cc}\hfill {\mathbf{P}}_1& =\mathbf{P}\left(\left|{\displaystyle \frac{\left(p-\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(Y_i=1)\right)\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=1)}{p\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(Y_i=1)}}\right|>{\displaystyle \frac{g}{2}}\right)\hfill \\ & =\mathbf{P}\left(\left|p-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathbf{I}(Y_i=1)\right|>{\displaystyle \frac{gp\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(Y_i=1)}{2\frac{1}{n}{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=1)}},\overline{A},\overline{B}\right)+\mathbf{P}\left(A\right)+\mathbf{P}\left(B\right)\hfill \\ & \le \mathbf{P}\left(\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|>{\displaystyle \frac{gp}{8p_j}}\right)+\mathbf{P}\left(A\right)+\mathbf{P}\left(B\right).\hfill \end{array}$$

(20)

Next, for any $1\le j\le r$, inequality (13) provides the following upper bound for the first term from the righthand side of the last inequality above (20),

$$\mathbf{P}\left(\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|>{\displaystyle \frac{gp}{8p_j}}\right)\le 2exp\{-{\displaystyle \frac{n{g}^2{p}^2}{2^7{(max(p,q))}^2{\left(\underset{1\le j\le r}{max}{p}_j\right)}^2}}\}.$$

(21)

The upper bound for the probabilities $\mathbf{P}\left(A\right)$, $\mathbf{P}\left(B\right)$, and ${\mathbf{P}}_2$ are obtained by the Hoeffding inequality:

$$\begin{array}{cc}\hfill \mathbf{P}\left(A\right)& =\mathbf{P}\left(\sum _{i=1}^n(\mathbf{I}(X_i=a_j,Y_i=1)-p{p}_j)>np{p}_j\right)\hfill \\ & \le exp\left\{-{\displaystyle \frac{n{p}^2{\left(\underset{1\le j\le r}{min}{p}_j\right)}^2}{2{\left(\underset{1\le j\le r}{max}(max(p{p}_j,1-p{p}_j))\right)}^2}}\right\},\hfill \\ \hfill \mathbf{P}\left(B\right)& =\mathbf{P}\left(\sum _{i=1}^n(p-\mathbf{I}(Y_i=1))>{\displaystyle \frac{pn}{2}}\right)\le exp\left\{-{\displaystyle \frac{n{p}^2}{8{(max(p,q))}^2}}\right\},\hfill \\ \hfill {\mathbf{P}}_2& \le 2exp\left\{-{\displaystyle \frac{n{g}^2{p}^2}{8{\left(\underset{1\le j\le r}{max}(max(p{p}_j,1-p{p}_j))\right)}^2}}\right\}.\hfill \end{array}$$

(22)

Finally, inequalities (19)–(22) yield relation (17).

The proof of inequality (18) has a similar structure as the proof of (17). □

The next lemma contains a huge, multilevel upper bound, which was used in the proof of Theorem 2.

Lemma 2. 

For any $g>0$, the following inequality holds

$$\begin{array}{cc}\hfill \underset{1\le j\le r}{max}& \mathbf{P}\left(\left|ln\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}\right)\right|>g\right)\hfill \\ & \le 4exp\left\{-{\displaystyle \frac{n{g}^2{p}^2{p}_{min}^2}{2^{11}{(max(p,q))}^2{p}_{max}^2}}\right\}+4exp\left\{-{\displaystyle \frac{n{g}^2{p}^2{p}_{min}^2}{2^7{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}\hfill \\ & +2exp\left\{-{\displaystyle \frac{n{p}^2{p}_{min}^2}{2^9{(max(p,q))}^2{p}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{p}^2{p}_{min}^2}{2^5{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}\hfill \\ & +3exp\left\{-{\displaystyle \frac{n{p}^2{p}_{min}^2}{2{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}+3exp\left\{-{\displaystyle \frac{n{p}^2}{8{(max(p,q))}^2}}\right\}\hfill \\ & +4exp\left\{-{\displaystyle \frac{n{g}^2{q}^2{q}_{min}^2}{2^{11}{(max(p,q))}^2{q}_{max}^2}}\right\}+4exp\left\{-{\displaystyle \frac{n{g}^2{q}^2{q}_{min}^2}{2^7{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}\hfill \\ & +2exp\left\{-{\displaystyle \frac{n{q}^2{q}_{min}^2}{2^9{(max(p,q))}^2{q}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{q}^2{q}_{min}^2}{2^5{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}\hfill \\ & +3exp\left\{-{\displaystyle \frac{n{q}^2{q}_{min}^2}{2{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}+3exp\left\{-{\displaystyle \frac{n{q}^2}{8{(max(p,q))}^2}}\right\}.\hfill \end{array}$$

(23)

Proof. 

For any $g>0$ and $1\le j\le r$, we obtain

$$\mathbf{P}\left(\left|ln\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}\right)\right|>g\right)=\mathbf{P}\left(ln\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}\right)>g\right)+\mathbf{P}\left(-ln\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}\right)>g\right)=:{\mathbf{P}}_1+{\mathbf{P}}_2.$$

(24)

Let us bound ${\mathbf{P}}_1$ from above. Using inequality $e^g-1\ge g$ and using Lemma 1, for any $1\le j\le r$, we obtain

$$\begin{array}{cc}\hfill {\mathbf{P}}_1=& \mathbf{P}\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}>e^g\right)=\mathbf{P}\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}-1>e^g-1\right)\le \mathbf{P}\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j}{p_j{\widehat{q}}_{j,n}}}-1>g\right)\hfill \\ \hfill \le & \mathbf{P}\left({\displaystyle \frac{{\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}}{p_j{\widehat{q}}_{j,n}}}>g,{\widehat{q}}_{j,n}\ge {\displaystyle \frac{q_j}{2}}\right)+\mathbf{P}\left({\widehat{q}}_{j,n}<{\displaystyle \frac{q_j}{2}}\right)\hfill \\ \hfill \le & \mathbf{P}\left({\widehat{p}}_{j,n}{q}_j-p_j{\widehat{q}}_{j,n}>{\displaystyle \frac{g_1{p}_j{q}_j}{2}}\right)+\mathbf{P}\left({\widehat{q}}_{j,n}<{\displaystyle \frac{q_j}{2}}\right)\hfill \\ \hfill =& \mathbf{P}\left(q_j({\widehat{p}}_{j,n}-p_j)-p_j({\widehat{q}}_{j,n}-q_j)>{\displaystyle \frac{g{p}_j{q}_j}{2}}\right)+\mathbf{P}\left({\widehat{q}}_{j,n}<{\displaystyle \frac{q_j}{2}}\right)\hfill \\ \hfill \le & \mathbf{P}\left(q_j|{\widehat{p}}_{j,n}-p_j|+p_j|{\widehat{q}}_{j,n}-q_j|>{\displaystyle \frac{g{p}_j{q}_j}{2}}\right)+\mathbf{P}\left({\widehat{q}}_{j,n}<{\displaystyle \frac{q_j}{2}}\right)\hfill \\ \hfill \le & \mathbf{P}\left(|{\widehat{p}}_{j,n}-p_j|>{\displaystyle \frac{g{p}_j}{4}}\right)+\mathbf{P}\left(|{\widehat{q}}_{j,n}-q_j|>{\displaystyle \frac{g{q}_j}{4}}\right)+\mathbf{P}\left({\widehat{q}}_{j,n}<{\displaystyle \frac{q_j}{2}}\right)\hfill \\ \hfill =& \mathbf{P}\left(|{\widehat{p}}_{j,n}-p_j|>{\displaystyle \frac{g{p}_j}{4}}\right)+\mathbf{P}\left(|{\widehat{q}}_{j,n}-q_j|>{\displaystyle \frac{g{q}_j}{4}}\right)+\mathbf{P}\left(q_j-{\widehat{q}}_{j,n}>{\displaystyle \frac{q_j}{2}}\right)\hfill \end{array}$$

Next, we apply the upper bounds (17) and (18) for the probabilities in the last line from the inequality above. Thus,

$$\begin{array}{cc}\hfill {\mathbf{P}}_1& \le 2exp\left\{-{\displaystyle \frac{n{g}^2{p}^2{p}_{min}^2}{2^{11}{(max(p,q))}^2{p}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{g}^2{p}^2{p}_{min}^2}{2^7{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}\hfill \\ & +exp\left\{-{\displaystyle \frac{n{p}^2{p}_{min}^2}{2{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}+exp\left\{-{\displaystyle \frac{n{p}^2}{8{(max(p,q))}^2}}\right\}\hfill \\ & +2exp\left\{-{\displaystyle \frac{n{g}^2{q}^2{q}_{min}^2}{2^{11}{(max(p,q))}^2{q}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{g}^2{q}^2{q}_{min}^2}{2^7{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}\hfill \\ & +2exp\left\{-{\displaystyle \frac{n{q}^2{q}_{min}^2}{2^9{(max(p,q))}^2{q}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{q}^2{q}_{min}^2}{2^5{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}\hfill \\ & +2exp\left\{-{\displaystyle \frac{n{q}^2{q}_{min}^2}{2{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{q}^2}{8{(max(p,q))}^2}}\right\}.\hfill \end{array}$$

(25)

We obtain the upper bound for ${\mathbf{P}}_2$ in the same way:

$$\begin{array}{cc}\hfill {\mathbf{P}}_2& \le 2exp\left\{-{\displaystyle \frac{n{g}^2{p}^2{p}_{min}^2}{2^{11}{(max(p,q))}^2{p}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{g}^2{p}^2{p}_{min}^2}{2^7{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}\hfill \\ & +2exp\left\{-{\displaystyle \frac{n{p}^2{p}_{min}^2}{2^9{(max(p,q))}^2{p}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{p}^2{p}_{min}^2}{2^5{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}\hfill \\ & +2exp\left\{-{\displaystyle \frac{n{p}^2{p}_{min}^2}{2{(max(p{p}_{max},1-p{p}_{min}))}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{p}^2}{8{(max(p,q))}^2}}\right\}\hfill \\ & +2exp\left\{-{\displaystyle \frac{n{g}^2{q}^2{q}_{min}^2}{2^{11}{(max(p,q))}^2{q}_{max}^2}}\right\}+2exp\left\{-{\displaystyle \frac{n{g}^2{q}^2{q}_{min}^2}{2^7{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}\hfill \\ & +exp\left\{-{\displaystyle \frac{n{q}^2{q}_{min}^2}{2{(max(q{q}_{max},1-q{q}_{min}))}^2}}\right\}+exp\left\{-{\displaystyle \frac{n{q}^2}{8{(max(p,q))}^2}}\right\}.\hfill \end{array}$$

(26)

Inequalities (24)–(26) imply (23). □

Lemma 3. 

For any given constants $b_j$, $c_j$, $1\le j\le r$, such that ${\displaystyle \sum _{j=1}^r}\left(\right|b_j|+|c_j\left|\right)>0$, the following convergence takes place

$$\sqrt{n}\sum _{j=1}^r\left(({\widehat{p}}_{j,n}-p_j)b_j+({\widehat{q}}_{j,n}-q_j)c_j\right)\stackrel{d}{\to }\xi \sim N(0,{\sigma }^2),asn\to \infty ,$$

with

$$\begin{array}{cc}\hfill {\sigma }^2:=\mathbf{E}\left(\sum _{j=1}^r\right(& \left({\displaystyle \frac{1}{p}}(\mathbf{I}(X=a_j,Y=1)-p{p}_j)-p_j(\mathbf{I}(Y=1)-p)\right)b_j\hfill \\ & +\left({\displaystyle \frac{1}{q}}(\mathbf{I}(X=a_j,Y=0)-q{q}_j)-q_j(\mathbf{I}(Y=0)-q)\right)c_j){)}^2.\hfill \end{array}$$

Proof. 

Denote

$$\begin{array}{cc}\hfill y_{j,n}& :={\displaystyle \frac{1}{\sqrt{n}p}}\sum _{i=1}^n(\mathbf{I}(X_i=a_j,Y_i=1)-p{p}_j)-{\displaystyle \frac{p_j}{\sqrt{n}}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p),\hfill \\ \hfill z_n& :={\displaystyle \frac{1}{np}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p),v_{j,n}:={\displaystyle \frac{1}{\sqrt{n}p}}\sum _{i=1}^n(\mathbf{I}(X_i=a_j,Y_i=1)-p{p}_j).\hfill \end{array}$$

We have

$$\begin{array}{cc}\hfill |\sqrt{n}& ({\widehat{p}}_{j,n}-p_j)-y_{j,n}|=\left|{\displaystyle \frac{\frac{1}{\sqrt{n}}{\displaystyle \sum _{i=1}^n}\mathbf{I}(X_i=a_j,Y_i=1)}{p\left(1+\frac{1}{np}{\displaystyle \sum _{i=1}^n}(\mathbf{I}(Y_i=1)-p)\right)}}-\sqrt{n}{p}_j-y_{j,n}\right|\hfill \\ \hfill =& \left|{\displaystyle \frac{1}{\sqrt{n}p}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)\left(1-z_n+\sum _{k=2}^{\infty }{(-z_n)}^k\right)-\sqrt{n}{p}_j-y_{j,n}\right|\mathbf{I}\left(|z_n|\le {\displaystyle \frac{1}{2}}\right)\hfill \\ & +|\sqrt{n}({\widehat{p}}_{j,n}-p_j)-y_{j,n}|\mathbf{I}\left(|z_n|>{\displaystyle \frac{1}{2}}\right)\hfill \\ \hfill =& \left|(v_{j,n}+\sqrt{n}{p}_j)\left(1-z_n+\sum _{k=2}^{\infty }{(-z_n)}^k\right)-\sqrt{n}{p}_j-y_{j,n}\right|\mathbf{I}\left(|z_n|\le {\displaystyle \frac{1}{2}}\right)\hfill \\ & +|\sqrt{n}({\widehat{p}}_{j,n}-p_j)-y_{j,n}|\mathbf{I}\left(|z_n|>{\displaystyle \frac{1}{2}}\right)\hfill \\ \hfill =& \left|-v_{j,n}{z}_n+{\displaystyle \frac{1}{\sqrt{n}p}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)\sum _{k=2}^{\infty }{(-z_n)}^k\right|\mathbf{I}\left(|z_n|\le {\displaystyle \frac{1}{2}}\right)\hfill \\ & +|\sqrt{n}({\widehat{p}}_{j,n}-p_j)-y_{j,n}|\mathbf{I}\left(|z_n|>{\displaystyle \frac{1}{2}}\right)\hfill \\ \hfill \le & |v_{j,n}{z}_n|\mathbf{I}\left(|z_n|\le {\displaystyle \frac{1}{2}}\right)+\left|{\displaystyle \frac{2}{\sqrt{n}p}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)z_n^2\right|\mathbf{I}\left(|z_n|\le {\displaystyle \frac{1}{2}}\right)\hfill \\ & +|\sqrt{n}({\widehat{p}}_{j,n}-p_j)-y_{j,n}|\mathbf{I}\left(|z_n|>{\displaystyle \frac{1}{2}}\right)\mathrm{a}.\mathrm{s}.\hfill \end{array}$$

(27)

Utilizing (27), for any $1\le j\le r$, $\epsilon >0$, we obtain

$$\begin{array}{cc}\hfill \mathbf{P}(|\sqrt{n}& ({\widehat{p}}_{j,n}-p_j)-y_{j,n}|>\epsilon )\hfill \\ \hfill \le & \mathbf{P}\left(|v_{j,n}{z}_n|\mathbf{I}\left(|z_n|\le {\displaystyle \frac{1}{2}}\right)>{\displaystyle \frac{\epsilon }{3}}\right)\hfill \\ & +\mathbf{P}\left(\left|{\displaystyle \frac{2}{\sqrt{n}p}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)z_n^2\right|\mathbf{I}\left(|z_n|\le {\displaystyle \frac{1}{2}}\right)>{\displaystyle \frac{\epsilon }{3}}\right)\hfill \\ & +\mathbf{P}\left(|\sqrt{n}({\widehat{p}}_{j,n}-p_j)-y_{j,n}|\mathbf{I}\left(|z_n|>{\displaystyle \frac{1}{2}}\right)>{\displaystyle \frac{\epsilon }{3}}\right)\hfill \\ \hfill \le & \mathbf{P}\left(|v_{j,n}{z}_n|>{\displaystyle \frac{\epsilon }{3}}\right)+\mathbf{P}\left(\left|{\displaystyle \frac{2}{\sqrt{n}p}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)z_n^2\right|>{\displaystyle \frac{\epsilon }{3}}\right)+\mathbf{P}\left(|z_n|>{\displaystyle \frac{1}{2}}\right)\hfill \\ \hfill =:& {\mathbf{P}}_1+{\mathbf{P}}_2+{\mathbf{P}}_3.\hfill \end{array}$$

(28)

We obtain the upper bound for ${\mathbf{P}}_1$ using inequalities from Lemma 1. Indeed, inequalities (13) and (15)) imply that for any $\epsilon >0$, there exists $C>0$ such that

$$\begin{array}{cc}\hfill {\mathbf{P}}_1=& \mathbf{P}\left(\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(X_i=a_j,Y_i=1)-p{p}_j)\right|·\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|>{\displaystyle \frac{p^2\epsilon }{3\sqrt{n}}}\right)\hfill \\ \hfill \le & \mathbf{P}\left(\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(X_i=a_j,Y_i=1)-p{p}_j)\right|>{\displaystyle \frac{p\sqrt{\epsilon }}{\sqrt{3}{n}^{\frac{1}{4}}}}\right)\hfill \\ & +\mathbf{P}\left(\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|>{\displaystyle \frac{p\sqrt{\epsilon }}{\sqrt{3}{n}^{\frac{1}{4}}}}\right)\le 2exp\left\{-C\sqrt{n}\right\}.\hfill \end{array}$$

(29)

We obtain the upper bound for the probability ${\mathbf{P}}_2$ in the same way. Indeed, inequalities (13) and (15) imply that for any $\epsilon >0$, there exists $C>0$ such that

$$\begin{array}{cc}\hfill {\mathbf{P}}_2=& \mathbf{P}\left(\left({\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)\right){\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|}^2>{\displaystyle \frac{p^3\epsilon }{6\sqrt{n}}}\right)\hfill \\ \hfill \le & \mathbf{P}\left({\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|}^2>{\displaystyle \frac{p^2\epsilon }{3\sqrt{n}{p}_j}},{\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)\ge {\displaystyle \frac{p{p}_j}{2}}\right)\hfill \\ & +\mathbf{P}\left({\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbf{I}(X_i=a_j,Y_i=1)<{\displaystyle \frac{p{p}_j}{2}}\right)\hfill \\ \hfill \le & \mathbf{P}\left(\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|>{\displaystyle \frac{p\sqrt{\epsilon }}{\sqrt{3p_j}{n}^{\frac{1}{4}}}}\right)+\mathbf{P}\left({\displaystyle \frac{1}{n}}\left|\sum _{i=1}^n(p{p}_j-\mathbf{I}(X_i=a_j,Y_i=1))\right|>{\displaystyle \frac{p{p}_j}{2}}\right)\hfill \\ \hfill \le & exp\left\{-C\sqrt{n}\right\}+exp\left\{-Cn\right\}.\hfill \end{array}$$

(30)

Finally, the bound for the probability ${\mathbf{P}}_3$ is provided by inequality (13):

$${\mathbf{P}}_3=\mathbf{P}\left(\left|{\displaystyle \frac{1}{n}}\sum _{i=1}^n(\mathbf{I}(Y_i=1)-p)\right|>{\displaystyle \frac{1}{2}}\right)\le exp\left\{-Cn\right\},$$

(31)

for some $C>0$. Relations (28)–(31) imply that for any $\epsilon >0$, there exists $C>0$ such that

$$\underset{1\le j\le r}{max}\mathbf{P}\left(\right|\sqrt{n}({\widehat{p}}_{j,n}-p_j)-y_{j,n}|>\epsilon )\le 4exp\left\{-C\sqrt{n}\right\}.$$

(32)

Denote

$$y_{j,n}^{\prime }:={\displaystyle \frac{1}{\sqrt{n}q}}\sum _{i=1}^n(\mathbf{I}(X_i=a_j,Y_i=0)-q{q}_j)-{\displaystyle \frac{q_j}{\sqrt{n}}}\sum _{i=1}^n(\mathbf{I}(Y_i=0)-q).$$

In a completely similar manner to the above, for any $\epsilon >0$ and some $C>0$, we obtain

$$\underset{1\le j\le r}{max}\mathbf{P}\left(\right|\sqrt{n}({\widehat{q}}_{j,n}-q_j)-y_{j,n}^{\prime }|>\epsilon )\le 4exp\left\{-C\sqrt{n}\right\}.$$

(33)

It is easy to see that

$$\begin{array}{cc}\hfill \sqrt{n}& \sum _{j=1}^r\left(({\widehat{p}}_{j,n}-p_j)b_j+({\widehat{q}}_{j,n}-q_j)c_j\right)\hfill \\ & =\sum _{j=1}^r\left(y_{j,n}{b}_j+y_{j,n}^{\prime }{c}_j\right)+\sum _{j=1}^r(\sqrt{n}({\widehat{p}}_{j,n}-p_j)-y_{j,n})b_j+\sum _{j=1}^r(\sqrt{n}({\widehat{q}}_{j,n}-q_j)-y_{j,n}^{\prime })c_j\hfill \\ & =:{\xi }_n+{\varphi }_n+{\zeta }_n.\hfill \end{array}$$

By (32) and (33), for any $\epsilon >0$, we obtain

$$\underset{n\to \infty }{lim}\mathbf{P}\left(\right|{\varphi }_n+{\zeta }_n|>\epsilon )=0.$$

Therefore, by Slutsky’s theorem, the weak limit of the original sequence coincides with the limit sequences ${\xi }_n$. It remains to be noted that

$$\begin{array}{cc}\hfill {\xi }_n={\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^r}(& \left({\displaystyle \frac{1}{p}}(\mathbf{I}(X_i=a_j,Y_i=1)-p{p}_j)-p_j(\mathbf{I}(Y_i=1)-p)\right)b_j\hfill \\ & +\left({\displaystyle \frac{1}{q}}(\mathbf{I}(X_i=a_j,Y_i=0)-q{q}_j)-q_j(\mathbf{I}(Y_i=0)-q)\right)c_j)\hfill \end{array}$$

and to apply the central limit theorem. □

4. Conclusions

In this paper, we studied the symmetrized version of the KL divergence, ${\mathcal{D}}_{KL}^{sym}\left(\mathbf{p}\right|\left|\mathbf{q}\right):={\mathcal{D}}_{KL}\left(\mathbf{p}\right|\left|\mathbf{q}\right)+{\mathcal{D}}_{KL}\left(\mathbf{q}\right|\left|\mathbf{p}\right)$. Today, it is known as Jeffreys divergence and is popular for classification problems. In this context, the distributions $\mathbf{p}$ and $\mathbf{q}$ represent the conditional probability distributions of a characteristic of interest under two different classes or conditions.

We established the asymptotic unbiasedness and normality of the plug-in estimator for the Jeffreys divergence. We considered a one-sample estimator, where, for a given sample size n, the number of observations in one condition is random and follows a binomial distribution. This differs from the traditional approach, where the properties of estimators are studied as the given sample sizes n and m of the two classes increase (see [38]). The results were expected, but additional technical work was required due to the randomness of the number of observations in one class. Moreover, we did not find detailed proofs for the Jeffreys divergence in the literature.

In this paper, we avoided referencing some known methods for proving normality, such as the $\delta$-method, and provided detailed proofs instead. We believe that such proofs are accessible to undergraduate students.