Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

1. Introduction

Many machine learning algorithms for classification and clustering employ a variety of dissimilarity measures. Information theory, convex analysis, and information geometry play key roles in the formulation of such divergences [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25].

The most popular and often used are: Squared Euclidean distance and Kullback–Leibler divergence. Recently, alternative generalized divergences such as the Csiszár–Morimoto f-divergence and Bregman divergence become attractive alternatives for advanced machine learning algorithms [26,27,28,29,30,31,32,33,34]. In this paper, we discuss a robust parameterized subclass of the Csiszár–Morimoto and the Bregman divergences: Alpha- and Beta-divergences that may provide more robust solutions with respect to outliers and additive noise and improved accuracy. Moreover, we provide links to new-class of robust Gamma-divergences [35] and extend this class to so called Alpha-Gamma divergences.

Divergences are considered here as (dis)similarity measures. Generally speaking, they measure a quasi-distance or directed difference between two probability distributions $\mathbf{P}$ and $\mathbf{Q}$, which can also be expressed for unconstrained nonnegative multi-way arrays and patterns.

In this paper we assume that $\mathbf{P}$ and $\mathbf{Q}$ are positive measures (densities) not necessary normalized, but should be finite measures. In the special case of normalized densities, we explicitly refer to these as probability densities. If we do not mention explicitly we assume that these measures are continuous. An information divergence is a measure of distance between two probability curves. In this paper, we discuss only one-dimensional probability curves (represented by nonnegative signals or time series). Generalization to two or multidimensional dimensional variables is straightforward. One density $\mathbf{Q}(x)$ is usually known and fixed and another one $\mathbf{P}(\mathit{x})$ is learned or adjusted to achieve a best in some sense similarity to the $\mathbf{Q}(\mathit{x})$. For example, a discrete density $\mathbf{Q}$ corresponds to the observed data and the vector $\mathbf{P}$ to be estimated, or expected data which are subject to constraints imposed on the assumed models. For the Non-negative Matrix Factorization (NMF) problem $\mathbf{Q}$ corresponds to the data matrix $\mathbf{Y}$ and $\mathbf{P}$ corresponds to estimated matrix $\widehat{\mathbf{Y}}=\mathbf{A}\mathbf{X}$ (or vice versa) [30].

The distance between two densities is called a metric if the following conditions hold:

• $D(\mathbf{P}||\mathbf{Q})\ge 0$ with equality if and only if $\mathbf{P}=\mathbf{Q}$ (nonnegativity and positive definiteness),
• $D(\mathbf{P}||\mathbf{Q})=D(\mathbf{Q}||\mathbf{P})$ (symmetry),
• $D(\mathbf{P}||\mathbf{Z})\le D(\mathbf{P}||\mathbf{Q})+D(\mathbf{Q}||\mathbf{Z})$ (subaddivity/triangle inequality).

Distances which only satisfies Condition 1 are not a metric and are referred to as (asymmetric) divergences.

In many applications, such as image analysis, pattern recognition and statistical machine learning we use the information-theoretic divergences rather than Euclidean squared or $l_p$-norm distances [28]. Several information divergences such as Kullback–Leibler, Hellinger and Jensen–Shannon divergences are central to estimate similarity between distributions and have long history in information geometry.

The concept of a divergence is not restricted to Euclidean spaces but can be extended to abstract spaces with the help of Radon–Nikodym derivative (see for example [36]). Let $(\mathbf{X},\mathcal{A},\mu )$ be a measure space, where μ is a finite or a σ-finite measure on $(\mathbf{X},\mathcal{A})$ and let assume that $\mathbf{P}$ and $\mathbf{Q}$ are two (probability) measures on $(\mathbf{X},\mathcal{A})$ such that $\mathbf{P}<<\mu$, $\mathbf{Q}<<\mu$ are absolutely continuous with respect to a measure μ, e.g., $\mu =\mathbf{P}+\mathbf{Q}$ and that $p=\frac{d\mathbf{P}}{d\mu }$ and $q=\frac{d\mathbf{Q}}{d\mu }$ the (densities) Radon–Nikodym derivative of $\mathbf{P}$ and $\mathbf{Q}$ with respect to μ. Using such notations the fundamental Kullback–Leibler (KL) divergence between two probabilities distributions can be written as

$$D_{KL}(\mathbf{P}||\mathbf{Q})={\int }_{\mathbf{X}}p(\mathit{x})log\left(\frac{p(\mathit{x})}{q(\mathit{x})}\right)d\mu (\mathit{x})$$

(1)

which is related to the Shannon entropy

$$H_S(\mathbf{P})=-{\int }_{\mathbf{X}}p(\mathit{x})log(p(\mathit{x}))d\mu (\mathit{x})$$

(2)

via

$$D_{KL}(\mathbf{P}||\mathbf{Q})=V_S(\mathbf{P},\mathbf{Q})-H_S(\mathbf{P})$$

where

$$V_S(\mathbf{P},\mathbf{Q})=-{\int }_{\mathbf{X}}p(\mathit{x})log(q(\mathit{x}))d\mu (\mathit{x})$$

is the Shannon’s cross entropy, provided that integrals exist. (In measure theoretic terms, the integral exists if the measure induced by $\mathbf{P}$ is absolutely continuous with respect to that induced by $\mathbf{Q}$). Here and in the whole paper we assume that all integrals exist.

The Kullback–Leibler divergence has been generalized by using a family of functions called generalized logarithm functions or α-logarithm

$${log}_{\alpha }(x)=\frac{1}{1-\alpha }(x^{1-\alpha }-1)(\mathrm{for}x>0)$$

(3)

which is a power function of x with power $1-\alpha$, and is the natural logarithm function in the limit $\alpha \to 1$. Often, the power function (3) allows to generate more robust divergences in respect to outliers and consequently better or more flexible performance (see, for example, [37]).

By using this type of extension, we derive and review three series of divergences, the Alpha, Beta- and Gamma- divergences all of which are generalizations of the KL-divergence. Moreover, we show its relation to the Tsallis entropy and the Rényi entropy (see Appendix A and Appendix B). It will be also shown how the Alpha-divergences are derived from the Csiszár–Morimoto f-divergence and the Beta-divergence form the Bregman divergence by using the power functions.

Similarly to work of Zhang [22,23,24] and Hein and Bousquet [21] one of our motivations is to show the close links and relations among wide class of divergences and provide an elegant way to handle the known divergences and intermediate and new one in the same framework. However, our similarity measures are different from these proposed in [21] and our approach and results are quite different to these presented in [22].

It should be mentioned that there has been previous attempts of unifying divergence functions, (especially related to Alpha-divergence) starting from the work by Zhu and Rohwer [10,11], Amari and Nagaoka [3], Taneya [38,39], Zhang ([22] and Gorban and Judge [40]. In particular, Zhang in ([22], and in subsequent works [23,24] investigated the deep link between information geometry and various divergence functions mostly related to Alpha-divergence through a unified approach based on convex functions. However, the previous works have not considered explicitly links and relationships among ALL three fundamental classes (Alpha-, Beta, Gamma-) divergences. Moreover, some their basic properties are reviewed and extended.

The scope of the results presented in this paper is vast since the class of generalized (flexible) divergence functions include a large number of useful loss functions containing those based on the relative entropies, generalized Kullback–Leibler or I-divergence, Hellinger distance, Jensen–Shannon divergence, J-divergence, Pearson and Neyman Chi-square divergences, Triangular Discrimination and Arithmetic-Geometric divergence. Moreover, we show that some new divergences can be generated. Especially, we generate a new family of Alpha-Gamma divergences and Itakura–Saito like distances with the invariant scaling property which belongs to a wider class of Beta-divergences. Generally, these new scale-invariant divergences provide extension of the families of Beta- and Gamma- divergences. The discussed in this paper divergence functions are flexible because they allow us to generate a large number of well known and often used particular divergences (for specific values of tuning parameters). Moreover, by adjusting adaptive tuning parameters, we can optimize cost functions for learning algorithms and estimate desired parameters of a model in presence of noise and outliers. In other words, the discussed in this paper divergences, especially Beta- and Gama- divergences are robust in respect to outliers for some values of tuning parameters.

One of important features of the considered family of divergences is that they can give some guidance for the selection and even development of new divergence measures if necessary and allows to unify these divergences under the same framework using the Csiszár-Morimoto and Bregamn divergences and their fundamental properties. Moreover, these families of divergences are generally defined on unnormalized finite measures (not necessary normalized probabilities). This allows us to analyze patterns of different size to be weighted differently, e.g., images with different sizes or documents of different length. Such measures play also an important role in the areas of neural computation, pattern recognition, learning, estimation, inference, and optimization. We have already successfully applied a subset of such divergences as cost functions (possibly with additional constraints and regularization terms) to derive novel multiplicative and additive projected gradient algorithms for nonnegative matrix and tensor factorizations [30,31].

The divergences are closely related to the invariant geometrical properties of the manifold of probability distributions [5,6,7].

2. Family of Alpha-Divergences

The Alpha-divergences can be derived from the Csiszár–Morimoto f-divergence and as shown recently by Amari using some tricks also from the Bregman divergence [3,6] (see also Appendix A). The Alpha-divergence was proposed by Chernoff [41] and have been extensively investigated and extended by Amari [1,3,5,6] and other researchers. For some modifications and/or extensions see for example works of Liese and Vajda [36], Minka [42], Taneja [38,39,43], Cressie–Read [44], Zhu–Rohwer [10,11] and Zhang [22,23,24,25]. One of our motivation to investigate and explore the family of Alpha-divergences is to develop flexible and efficient learning algorithms for nonnegative matrix and tensor factorizations which unify and extend existing algorithms including such algorithms as EMML (Expectation Maximization Maximum Likelihood) and ISRA (Image Space Reconstruction Algorithm) (see [30,45] and references therein).

2.1. Asymmetric Alpha-Divergences

The basic asymmetric Alpha-divergence can be defined as [3]:

$$D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }(\mathit{x})q^{1-\alpha }(\mathit{x})-\alpha p(\mathit{x})+(\alpha -1)q(\mathit{x})\right)d\mu (\mathit{x}),\alpha \in \mathbb{R}\setminus \{0,1\}$$

(4)

where $p(\mathit{x})$ and $q(\mathit{x})$ do not need to be normalized.

The Alpha-divergence can be expressed via a generalized KL divergence as follows

$$D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=D_{GKL}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=-\frac{1}{\alpha }\int \left(p{log}_{\alpha }\left(\frac{q}{p}\right)+p-q\right)d\mu (\mathit{x}),\alpha \in \mathbb{R}\setminus \{0,1\}$$

(5)

For discrete probability measures with mass functions $\mathbf{P}=[p_1,p_2,\dots ,p_n]$ and $\mathbf{Q}=[q_1,q_2,\dots ,q_n]$, the discrete Alpha-divergence is formulated as a separable divergence:

$$D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\sum _{i=1}^n{d}_A^{(\alpha )}(p_i||q_i)=\frac{1}{\alpha (\alpha -1)}\sum _{i=1}^n\left(p_i^{\alpha }{q}_i^{1-\alpha }-\alpha p_i+(\alpha -1)q_i\right),\alpha \in \mathbb{R}\setminus \{0,1\}$$

(6)

where $d_A^{(\alpha )}(p_i||q_i)=\left(p_i^{\alpha }{q}_i^{1-\alpha }-\alpha p_i+(\alpha -1)q_i\right)/\left(\alpha (\alpha -1)\right).$

Note that this form of Alpha-divergence differs slightly from the loss function given by [1], because it was defined only for probability distributions, i.e., under assumptions that $\int p(\mathit{x})d\mu (\mathit{x})=1$ and $\int q(\mathit{x})d\mu (\mathit{x})=1$. It was extended by Zhu and Rohwer in [10,11] (see also Amari and Nagaoka [3]) for positive measures, by incorporating additional terms. These terms are needed to allow de-normalized densities (positive measures), in the same way that the generalized Kullback–Leibler divergences (I-divergence).

Extending the Kullback–Leibler divergence into the family of Alpha-divergence is a crucial step for a unified view of a wide class of divergence functions, since it demonstrates that the nonnegativity of alpha-divergences may be viewed as arising not only from Jensen’s inequality but the arithmetic-geometric inequality. As has been pointed out by Jun Zhang this can be fully exploited as a consequence of the inequality of convex functions or more generally by his convex-based approach [22].

For normalized densities: $\overline{p}(\mathit{x})=p(\mathit{x})/\int p(\mathit{x})d\mu (\mathit{x})$ and $\overline{q}(\mathit{x})=q(\mathit{x})/\int q(\mathit{x})d\mu (\mathit{x})$ the Alpha-divergence simplifies to [1,44]:

$$D_A^{(\alpha )}(\overline{\mathbf{P}}||\overline{\mathbf{Q}})=\frac{1}{\alpha (\alpha -1)}\left(\int {\overline{p}}^{\alpha }(\mathit{x}){\overline{q}}^{1-\alpha }(\mathit{x})d\mu (\mathit{x})-1\right),\alpha \in \mathbb{R}\setminus \{0,1\}$$

(7)

and is related to the Tsallis divergence and the Tsallis entropy [46] (see Appendix A):

$$H_T^{(\alpha )}(\overline{\mathbf{P}})=\frac{1}{1-\alpha }\left(\int {\overline{p}}^{\alpha }(\mathit{x})d\mu (\mathit{x})-1\right)=-\int {\overline{p}}^{\alpha }(\mathit{x}){log}_{\alpha }\overline{p}(\mathit{x})d\mu (\mathit{x})$$

(8)

provided integrals on the right exist.

In fact, the Tsallis entropy was first defined by Havrda and Charvat in 1967 [47] and almost forgotten and rediscovered by Tsallis in 1988 [46] in different context (see Appendix A).

Various authors use the parameter α in different ways. For example, using the Amari notation, ${\alpha }_A$ with $\alpha =(1-{\alpha }_A)/2$, the Alpha-divergence takes the following form [1,2,3,10,11,22,23,24,25,44,48,49]:

$${\tilde{D}}_A^{({\alpha }_A)}(P̠||Q̠)=\frac{4}{1-{\alpha }_A^2}\int \left({\displaystyle \frac{1-{\alpha }_A}{2}p+\frac{1+{\alpha }_A}{2}q-p^{{\displaystyle \frac{1-{\alpha }_A}{2}}}{q}^{{\displaystyle \frac{1+{\alpha }_A}{2}}}}\right)d\mu (\mathit{x}),{\alpha }_A\in \mathbb{R}\setminus \{\pm 1\}$$

(9)

When α takes values from 0 to 1, ${\alpha }_A$ takes values from $-1$ to 1. The duality exists between ${\alpha }_A$ and $-{\alpha }_A$, in the sense that ${\tilde{D}}_A^{({\alpha }_A)}(\mathbf{P}||\mathbf{Q})={\tilde{D}}_A^{(-{\alpha }_A)}(\mathbf{Q}||\mathbf{P})$.

In the special cases for $\alpha =2,0.5,-1$, we obtain from (4) the well known Pearson Chi-square, Hellinger and inverse Pearson, also called the Neyman Chi-square distances, given respectively by

$$\begin{array}{ccc}\hfill D_A^{(2)}(\mathbf{P}||\mathbf{Q})& =\hfill & \hfill D_P(\mathbf{P}||\mathbf{Q})=\frac{1}{2}\int \frac{{(p(\mathit{x})-q(\mathit{x}))}^2}{q(\mathit{x})}d\mu (\mathit{x})\\ \hfill D_A^{(1/2)}(\mathbf{P}||\mathbf{Q})& =\hfill & \hfill 2D_H(\mathbf{P}||\mathbf{Q})=2\int {(\sqrt{p(\mathit{x})}-\sqrt{q(\mathit{x})})}^{2}d\mu (\mathit{x})\\ \hfill D_A^{(-1)}(\mathbf{P}||\mathbf{Q})& =\hfill & \hfill D_N(\mathbf{P}||\mathbf{Q})=\frac{1}{2}\int \frac{{(p(\mathit{x})-q(\mathit{x}))}^2}{p(\mathit{x})}d\mu (\mathit{x})\end{array}$$

(10)

For the singular values $\alpha =1$ and $\alpha =0$ the Alpha-divergences (4) have to be defined as limiting cases respectively for $\alpha \to 1$ and $\alpha \to 0$. When this limit is evaluated (using the L’Hôpital’s rule) for $\alpha \to 1$, we obtain the Kullback–Leibler divergence:

$$\begin{array}{cc}\hfill D_{KL}(\mathbf{P}||\mathbf{Q})& =\underset{\alpha \to 1}{lim}{D}_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\underset{\alpha \to 1}{lim}\int \frac{p^{\alpha }{q}^{1-\alpha }logp-p^{\alpha }{q}^{1-\alpha }logq-p+q}{2\alpha -1}d\mu (\mathit{x})\hfill \\ & =\int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x})\hfill \end{array}$$

(11)

with the conventions $0/0=0,0log(0)=0$ and $p/0=\infty$ for $p>0$.

From the inequality $plogp\ge p-1$, it follows that the KL I-divergence is nonnegative and achieves zero if and only if $\mathbf{P}=\mathbf{Q}$.

Similarly, for $\alpha \to 0$, we obtain the reverse Kullback–Leibler divergence:

$$D_{KL}(\mathbf{Q}||\mathbf{P})=\underset{\alpha \to 0}{lim}{D}_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\int \left(qlog\left(\frac{q}{p}\right)-q+p\right)d\mu (\mathit{x})$$

(12)

Hence, the Alpha-divergence can be evaluated in a more explicit form as

$$D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(q(\mathit{x})\left[{\left(\frac{p(\mathit{x})}{q(\mathit{x})}\right)}^{\alpha }-1\right]-\alpha [p(\mathit{x})-q(\mathit{x})]\right)d\mu (\mathit{x}),\alpha \ne 0,1}\hfill \\ \\ {\displaystyle \int \left(q(\mathit{x})log\frac{q(\mathit{x})}{p(\mathit{x})}+p(\mathit{x})-q(\mathit{x})\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ \\ {\displaystyle \int \left(p(\mathit{x})log\frac{p(\mathit{x})}{q(\mathit{x})}-p(\mathit{x})+q(\mathit{x})\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$$

(13)

In fact, the Alpha-divergence smoothly connects the I-divergence $D_{KL}(\mathbf{P}||\mathbf{Q})$ with the reverse I-divergence $D_{KL}(\mathbf{Q}||\mathbf{P})$ and passes through the Hellinger distance [50]. Moreover, it also smoothly connects the Pearson Chi-square and Neyman Chi-square divergences and passes through the I-divergences [10,11].

The Alpha-divergence is a special case of Csiszár–Morimoto f-divergence [17,19] proposed later by Ali and Silvey [20]). This class of divergences were also independently defined by Morimoto [51]. The Csiszár–Morimoto f-divergence is associated to any function $f(u)$ that is convex over $(0,\infty )$ and satisfies $f(1)=0$:

$$D_f(\mathbf{P}||\mathbf{Q})=\int q(\mathit{x})f\left(\frac{p(\mathit{x})}{q(\mathit{x})}\right)d\mu (\mathit{x})$$

(14)

We define $0f(0/0)=0$ and $0f(a/0)={lim}_{t\to 0}tf(a/t)={lim}_{u\to \infty }f(u)/u$. Indeed, assuming $f(u)=(u^{\alpha }-\alpha u+\alpha -1))/({\alpha }^2-\alpha )$ yields the formula (4).

The Csiszár–Morimoto f-divergence has many beautiful properties [6,17,18,20,22,36,52,53]:

• Nonnegativity: The Csiszár–Morimoto f-divergence is always nonnegative, and equal to zero if and only if probability densities $p(\mathit{x})$ and $q(\mathit{x})$ coincide. This follows immediately from the Jensens inequality (for normalized densities):

  $$D_f(\mathbf{P}||\mathbf{Q})=\int qf\left(\frac{p}{q}\right)d\mu (\mathit{x})\ge f\left(\int q(\frac{p}{q})d\mu (\mathit{x})\right)=f(1)=0.$$

  (15)
• Generalized entropy: It corresponds to a generalized f-entropy of the form:

  $$H_f(\mathbf{P})=-\int f(p(\mathit{x}))d\mu (\mathit{x})$$

  (16)

  for which the Shannon entropy is a special case for $f(p)=plog(p)$. Note that $H_f$ is concave while f is convex.
• Convexity: For any $0\le \lambda \le 1$

  $$D_f(\lambda {\mathbf{P}}_1+(1-\lambda ){\mathbf{P}}_2||\lambda {\mathbf{Q}}_1+(1-\lambda ){\mathbf{Q}}_2)\le \lambda D_f({\mathbf{P}}_1||{\mathbf{Q}}_1)+(1-\lambda )D_f({\mathbf{P}}_2||{\mathbf{Q}}_2).$$

  (17)
• Scaling: For any positive constant $c>0$ we have

  $$c{D}_f(\mathbf{P}||\mathbf{Q})=D_{cf}(\mathbf{P}||\mathbf{Q})$$

  (18)
• Invariance: The f-divergence is invariant to bijective transformations [6,20]. This means that, when $\mathit{x}$ is transformed to $\mathit{y}$ bijectively by

  $$\mathit{y}=\mathit{k}(\mathit{x})$$

  (19)

  probability distribution $\mathbf{P}(\mathit{x})$ changes to $\tilde{\mathbf{P}}(\mathit{y})$. However,

  $$D_f(\mathbf{P}||\mathbf{Q})=D_f(\tilde{\mathbf{P}}||\tilde{\mathbf{Q}})$$

  (20)

  Additionally,

  $$D_f(\mathbf{P}||\mathbf{Q})=D_{\tilde{f}}(\mathbf{P}||\mathbf{Q})$$

  (21)

  for $\tilde{f}(u)=f(u)-c(u-1)$ for an arbitrary constant c, and

  $$D_f(\mathbf{P}||\mathbf{Q})=D_{f^{*}}(\mathbf{Q}||\mathbf{P})$$

  (22)

  where $f^{*}(u)=uf(1/u)$ is called a conjugate function.
• Symmetricity: For an arbitrary Csiszár–Morimoto f-divergence, it is possible to construct a symmetric divergence for $f_{sym}(u)=f(u)+f^{*}(u)$.
• Boundedness The Csiszár–Morimoto f-divergence for positive measures (densities) is bounded (if limit exists and it is finite) [28,36]

  $$0\le D_f(\mathbf{P}||\mathbf{Q})\le \underset{u\to 0^{+}}{lim}\left\{f(u)+uf(\frac{1}{u})\right\}$$

  (23)

  Furthermore [54],

  $$0\le D_f(\mathbf{P}||\mathbf{Q})\le \int (p-q)f^{\prime }(\frac{p}{q})d\mu (\mathit{x})$$

  (24)

Using fundamental properties of the Csiszár–Morimoto f-divergence we can establish basic properties of the Alpha-divergences [3,6,22,40,42].

The Alpha-divergence (4) has the following basic properties:

• Convexity: $D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})$ is convex with respect to both $\mathbf{P}$ and $\mathbf{Q}$.
• Strict Positivity: $D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})\ge 0$ and $D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=0$ if and only if $\mathbf{P}=\mathbf{Q}$.
• Continuity: The Alpha-divergence is continuous function of real variable α in the whole range including singularities.
• Duality: $D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=D_A^{(1-\alpha )}(\mathbf{Q}||\mathbf{P})$.
• Exclusive/Inclusive Properties: [42]

  • For $\alpha \to -\infty$, the estimation of $q(\mathit{x})$ that approximates $p(\mathit{x})$ is exclusive, that is $q(\mathit{x})\le p(\mathit{x})$ for all $\mathit{x}$. This means that the minimization of $D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})$ with respect to $q(\mathit{x})$ will force $q(\mathit{x})$ to be exclusive approximation, i.e., the mass of $q(\mathit{x}$) will lie within $p(\mathit{x})$ (see detail and graphical illustrations in [42]).
  • For $\alpha \to \infty$, the estimation of $q(\mathit{x})$ that approximates $p(\mathit{x})$ is inclusive, that is $q(\mathit{x})\ge p(\mathit{x})$ for all $\mathit{x}$. In other words, the mass of $q(\mathit{x})$ includes all the mass of $p(\mathit{x})$.
• Zero-forcing and zero-avoiding properties: [42]
  Here, we treat the case where $p(\mathit{x})$ and $q(\mathit{x})$ are not necessary mutually absolutely continuous. In such a case the divergence may diverges to ∞. However, the following two properties hold:

  • For $\alpha \le 0$ the estimation of $q(\mathit{x})$ that approximates $p(\mathit{x})$ is zero-forcing (coercive), that is, $p(\mathit{x})=0$ forces $q(\mathit{x})=0$.
  • For $\alpha \ge 1$ the estimation of $q(\mathit{x})$ that approximates $p(\mathit{x})$ is zero-avoiding, that is, $p(\mathit{x})>0$ implies $q(\mathit{x})>0$.

One of the most important property of the Alpha-divergence is that it is a convex function with respect to $\mathbf{P}$ and $\mathbf{Q}$ and has a unique minimum for $\mathbf{P}=\mathbf{Q}$ (see e.g. [22]).

2.2. Alpha-Rényi Divergence

It is interesting to note that the Alpha-divergence is closely related to the Rényi divergence. We define an Alpha-Rényi divergence as

$$\begin{array}{cccc}\hfill D_{AR}^{(\alpha )}(\mathbf{P}||\mathbf{Q})& & \hfill =& \frac{1}{\alpha (\alpha -1)}log\left(1+\alpha (\alpha -1)D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q}\right)\hfill \\ & & \hfill =& \frac{1}{\alpha (\alpha -1)}log\left(\int \left(p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q\right)d\mu (\mathit{x})+1\right),\alpha \in \mathbb{R}\setminus \{0,1\}\hfill \end{array}$$

(25)

For $\alpha =0$ and $\alpha =1$ the Alpha-Rényi divergence simplifies to the Kullback–Leibler divergences:

$$\begin{array}{cccc}\hfill D_{KL}(\mathbf{P}||\mathbf{Q})=\underset{\alpha \to 1}{lim}{D}_{AR}^{(\alpha )}(\mathbf{P}||\mathbf{Q})& & \hfill =& \underset{\alpha \to 1}{lim}\frac{1}{2\alpha -1}\frac{\int (p^{\alpha }{q}^{1-\alpha }(logp-logq)-p+q)d\mu (\mathit{x})}{\int (p^{\alpha }{q}^{1-\alpha }-\alpha p-(\alpha -1)q)d\mu (\mathit{x})+1}\hfill \\ & & \hfill =& \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x})\hfill \end{array}$$

(26)

and

$$\begin{array}{c}\hfill D_{KL}(\mathbf{Q}||\mathbf{P})=\underset{\alpha \to 0}{lim}{D}_{AR}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\int \left(qlog\left(\frac{q}{p}\right)-q+p\right)d\mu (\mathit{x})\end{array}$$

(27)

We define the Alpha-Rényi divergence for normalized probability densities as follows:

$$D_{AR}^{(\alpha )}(\overline{\mathbf{P}}||\overline{\mathbf{Q}})=\frac{1}{\alpha (\alpha -1)}log\left(\int {\overline{p}}^{\alpha }{\overline{q}}^{1-\alpha }d\mu (\mathit{x})\right)=\frac{1}{\alpha -1}log{\left(\int \overline{q}{\left(\frac{\overline{p}}{\overline{q}}\right)}^{\alpha }d\mu (\mathit{x})\right)}^{\frac{1}{\alpha }}$$

(28)

which corresponds to the Rényi entropy [55,56,57] (see Appendix B)

$$H_R^{(\alpha )}(\overline{\mathbf{P}})=-\frac{1}{\alpha -1}log\left(\int {\overline{p}}^{\alpha }(\mathit{x})d\mu (\mathit{x})\right)$$

(29)

Note that we used different scaling factor than in the original Rényi divergence [56,57]. In general, the Alpha-Rényi divergence make sense only for the (normalized) probability densities (since, otherwise the function (25) can be complex-valued for some positive non-normalized measures).

Furthermore, the Alpha-divergence is convex with respect to positive densities p and q for any parameter of $\alpha \in \mathbb{R}$, while the Alpha-Rényi divergence (28) is convex jointly in p and q for $\alpha \in [0,1]$ and it is not convex for $\alpha >1$. Convexity implies that $D_{AR}^{(\alpha )}$ is increasing in α when p and q are fixed. Actually, the Rényi divergence is increasing in α on the whole set $(0,\infty )$ [58,59,60].

2.3. Extended Family of Alpha-Divergences

There are several ways to extend the asymmetric Alpha-divergence. For example, instead of q, we can take the average of q and p, under the assumption that if p and q are similar, they should be “close” to their average, so we can define the modified Alpha-divergence as

$$D_{Am1}^{(\alpha )}(\mathbf{P}||\tilde{\mathbf{Q}})=\frac{1}{\alpha (\alpha -1)}\int \left(\tilde{q}\left[{\left(\frac{p}{\tilde{q}}\right)}^{\alpha }-1\right]-\alpha (p-\tilde{q})\right)d\mu (\mathit{x})$$

(30)

whereas an adjoint Alpha-divergence is given by

$$D_{Am2}^{(\alpha )}(\tilde{\mathbf{Q}}||\mathbf{P})=\frac{1}{\alpha (\alpha -1)}\int \left(p\left[{\left(\frac{\tilde{q}}{p}\right)}^{\alpha }-1\right]+\alpha (p-\tilde{q})\right)d\mu (\mathit{x})$$

(31)

where $\tilde{\mathbf{Q}}=(\mathbf{P}+\mathbf{Q})/2$ and $\tilde{q}=(p+q)/2$.

For the singular values $\alpha =1$ and $\alpha =0$, the Alpha-divergences (30) can be evaluated as

$$\underset{\alpha \to 0}{lim}{D}_{Am1}^{(\alpha )}(\mathbf{P}||\tilde{\mathbf{Q}})=\int \left(\tilde{q}log\left(\frac{\tilde{q}}{p}\right)+p-\tilde{q}\right)d\mu (\mathit{x})=\int \left(\frac{p+q}{2}log\left(\frac{p+q}{2p}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x})$$

and

$$\underset{\alpha \to 1}{lim}{D}_{Am1}^{(\alpha )}(\mathbf{P}||\tilde{Q})=\int \left(plog\left(\frac{p}{\tilde{q}}\right)-p+\tilde{q}\right)d\mu (\mathit{x})=\int \left(plog\left(\frac{2p}{p+q}\right)+\frac{q-p}{2}\right)d\mu (\mathit{x})$$

As examples, we consider the following prominent cases for (31):

1. Triangular Discrimination (TD) [61]

$$D_{Am2}^{(-1)}(\tilde{\mathbf{Q}}||\mathbf{P})=\frac{1}{4}{D}_{TD}(\mathbf{P}||\mathbf{Q})=\frac{1}{4}\int \frac{{(p-q)}^2}{p+q}d\mu (\mathit{x})$$

(32)

2. Relative Jensen–Shannon divergence [62,63,64]

$$\underset{\alpha \to 0}{lim}{D}_{Am2}^{(\alpha )}(\tilde{\mathbf{Q}}||\mathbf{P})=D_{RJS}(\mathbf{P}||\mathbf{Q})=\int \left(plog\left(\frac{2p}{p+q}\right)-p+q\right)d\mu (\mathit{x})$$

(33)

3. Relative Arithmetic-Geometric divergence proposed by Taneya [38,39,43]

$$\underset{\alpha \to 1}{lim}{D}_{Am2}^{(\alpha )}(\tilde{\mathbf{Q}}||\mathbf{P})=\frac{1}{2}{D}_{RAG}(\mathbf{P}||\mathbf{Q})=\int \left((p+q)log\left(\frac{p+q}{2p}\right)+p-q\right)d\mu (\mathit{x})$$

(34)

3. Neyman Chi-square divergence

$$D_{Am2}^{(2)}(\tilde{\mathbf{Q}}||\mathbf{P})=\frac{1}{8}{D}_{{\chi }^2}(\mathbf{P}||\mathbf{Q})=\frac{1}{8}\int \frac{{(p-q)}^2}{p}d\mu (\mathit{x})$$

(35)

The asymmetric Alpha-divergences can be expressed formally as the Csiszár–Morimoto f-divergence, as shown in Table 1 [30].

Table 1. Asymmetric Alpha-divergences and associated convex Csiszár-Morimoto functions [30].

Divergence ${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\int q{f}^{(\alpha )}\left(\frac{p}{q}\right)d\mu (\mathit{x})}$	Csiszár function ${\displaystyle f^{(\alpha )}(u),u=p/q}$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(q\left[{\left(\frac{p}{q}\right)}^{\alpha }-1\right]-\alpha (p-q)\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\frac{q}{p}+p-q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(plog\frac{p}{q}-p+q\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{u^{\alpha }-1-\alpha (u-1)}{\alpha (\alpha -1)},\alpha \ne 0,1}\hfill \\ \\ {\displaystyle u-1-logu,\alpha =0,}\hfill \\ \\ {\displaystyle 1-u+ulogu,\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p\left[{\left(\frac{q}{p}\right)}^{\alpha }-1\right]+\alpha (p-q)\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(plog\frac{p}{q}-p+q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\frac{q}{p}+p-q\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{u^{1-\alpha }+(\alpha -1)u-\alpha }{\alpha (\alpha -1)},\alpha \ne 0,1,}\hfill \\ \\ {\displaystyle 1-u+ulogu,\alpha =0}\hfill \\ \\ {\displaystyle u-1-logu,\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(q{\left(\frac{p+q}{2q}\right)}^{\alpha }-q-\alpha \frac{p-q}{2}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{2q}{p+q}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p+q}{2q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{{\left({\displaystyle \frac{u+1}{2}}\right)}^{\alpha }-1-\alpha \left({\displaystyle \frac{u-1}{2}}\right)}{\alpha (\alpha -1)},\alpha \ne 0,1,}\hfill \\ {\displaystyle \frac{u-1}{2}+log\left(\frac{2}{u+1}\right),\alpha =0,}\hfill \\ {\displaystyle \frac{1-u}{2}+\frac{u+1}{2}log\left(\frac{u+1}{2}\right),\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p{\left(\frac{p+q}{2p}\right)}^{\alpha }-p+\alpha \frac{p-q}{2}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(plog\left(\frac{2p}{p+q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p+q}{2p}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{u{\left({\displaystyle \frac{u+1}{2u}}\right)}^{\alpha }-u-\alpha \left({\displaystyle \frac{1-u}{2}}\right)}{\alpha (\alpha -1)},\alpha \ne 0,1,}\hfill \\ {\displaystyle \frac{1-u}{2}-ulog\left({\displaystyle \frac{u+1}{2u}}\right),\alpha =0,}\hfill \\ {\displaystyle \frac{u-1}{2}+\left({\displaystyle \frac{u+1}{2}}\right)log\left({\displaystyle \frac{u+1}{2u}}\right),\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha -1}\int (p-q)\left[{\left(\frac{p+q}{2q}\right)}^{\alpha -1}-1\right]d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{(u-1)\left[{\left({\displaystyle \frac{u+1}{2}}\right)}^{\alpha -1}-1\right]}{\alpha -1},\alpha \ne 1,}\hfill \\ {\displaystyle (u-1)log\left(\frac{u+1}{2}\right),\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha -1}\int (q-p)\left[{\left(\frac{p+q}{2p}\right)}^{\alpha -1}-1\right]d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (q-p)log\left(\frac{p+q}{2p}\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{(1-u)\left[{\left({\displaystyle \frac{u+1}{2u}}\right)}^{\alpha -1}-1\right]}{\alpha -1},\alpha \ne 1,}\hfill \\ {\displaystyle (1-u)log\left({\displaystyle \frac{u+1}{2u}}\right),\alpha =1.}\hfill \end{array}\right.$

Table 1. Asymmetric Alpha-divergences and associated convex Csiszár-Morimoto functions [30].

Divergence ${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\int q{f}^{(\alpha )}\left(\frac{p}{q}\right)d\mu (\mathit{x})}$	Csiszár function ${\displaystyle f^{(\alpha )}(u),u=p/q}$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(q\left[{\left(\frac{p}{q}\right)}^{\alpha }-1\right]-\alpha (p-q)\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\frac{q}{p}+p-q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(plog\frac{p}{q}-p+q\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{u^{\alpha }-1-\alpha (u-1)}{\alpha (\alpha -1)},\alpha \ne 0,1}\hfill \\ \\ {\displaystyle u-1-logu,\alpha =0,}\hfill \\ \\ {\displaystyle 1-u+ulogu,\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p\left[{\left(\frac{q}{p}\right)}^{\alpha }-1\right]+\alpha (p-q)\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(plog\frac{p}{q}-p+q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\frac{q}{p}+p-q\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{u^{1-\alpha }+(\alpha -1)u-\alpha }{\alpha (\alpha -1)},\alpha \ne 0,1,}\hfill \\ \\ {\displaystyle 1-u+ulogu,\alpha =0}\hfill \\ \\ {\displaystyle u-1-logu,\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(q{\left(\frac{p+q}{2q}\right)}^{\alpha }-q-\alpha \frac{p-q}{2}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{2q}{p+q}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p+q}{2q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{{\left({\displaystyle \frac{u+1}{2}}\right)}^{\alpha }-1-\alpha \left({\displaystyle \frac{u-1}{2}}\right)}{\alpha (\alpha -1)},\alpha \ne 0,1,}\hfill \\ {\displaystyle \frac{u-1}{2}+log\left(\frac{2}{u+1}\right),\alpha =0,}\hfill \\ {\displaystyle \frac{1-u}{2}+\frac{u+1}{2}log\left(\frac{u+1}{2}\right),\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p{\left(\frac{p+q}{2p}\right)}^{\alpha }-p+\alpha \frac{p-q}{2}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(plog\left(\frac{2p}{p+q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p+q}{2p}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{u{\left({\displaystyle \frac{u+1}{2u}}\right)}^{\alpha }-u-\alpha \left({\displaystyle \frac{1-u}{2}}\right)}{\alpha (\alpha -1)},\alpha \ne 0,1,}\hfill \\ {\displaystyle \frac{1-u}{2}-ulog\left({\displaystyle \frac{u+1}{2u}}\right),\alpha =0,}\hfill \\ {\displaystyle \frac{u-1}{2}+\left({\displaystyle \frac{u+1}{2}}\right)log\left({\displaystyle \frac{u+1}{2u}}\right),\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha -1}\int (p-q)\left[{\left(\frac{p+q}{2q}\right)}^{\alpha -1}-1\right]d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{(u-1)\left[{\left({\displaystyle \frac{u+1}{2}}\right)}^{\alpha -1}-1\right]}{\alpha -1},\alpha \ne 1,}\hfill \\ {\displaystyle (u-1)log\left(\frac{u+1}{2}\right),\alpha =1.}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha -1}\int (q-p)\left[{\left(\frac{p+q}{2p}\right)}^{\alpha -1}-1\right]d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (q-p)log\left(\frac{p+q}{2p}\right)d\mu (\mathit{x}).}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{(1-u)\left[{\left({\displaystyle \frac{u+1}{2u}}\right)}^{\alpha -1}-1\right]}{\alpha -1},\alpha \ne 1,}\hfill \\ {\displaystyle (1-u)log\left({\displaystyle \frac{u+1}{2u}}\right),\alpha =1.}\hfill \end{array}\right.$

2.4. Symmetrized Alpha-Divergences

The basic Alpha-divergence is asymmetric, that is, $D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})\ne D_A^{(\alpha )}(\mathbf{Q}||\mathbf{P})$.

Generally, there are two ways to symmetrize divergences: Type-1

$$D_{S1}(\mathbf{P}||\mathbf{Q})=\frac{1}{2}\left[D_A(\mathbf{P}||\mathbf{Q})+D_A(\mathbf{Q}||\mathbf{P})\right]$$

(36)

and Type 2

$$D_{S2}(\mathbf{P}||\mathbf{Q})=\frac{1}{2}\left[D_A\left(\mathbf{P}||\frac{\mathbf{P}+\mathbf{Q}}{2}\right)+D_A\left(\mathbf{Q}||\frac{\mathbf{P}+\mathbf{Q}}{2}\right)\right]$$

(37)

The symmetric Alpha-divergence (Type-1) can be defined as (we will omit scaling factor $1/2$ for simplicity)

$$D_{AS1}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})+D_A^{(\alpha )}(\mathbf{Q}||\mathbf{P})=\int \frac{\left(p^{\alpha }-q^{\alpha }\right)\left(p^{1-\alpha }-q^{1-\alpha }\right)}{\alpha (1-\alpha )}d\mu (\mathit{x})$$

(38)

As special cases, we obtain several well-known symmetric divergences:

1. Symmetric Chi-Squared divergence [54]

$$D_{AS1}^{(-1)}(\mathbf{P}||\mathbf{Q})=D_{AS1}^{(2)}(\mathbf{P}||\mathbf{Q})=\frac{1}{2}{D}_{{\chi }^2}(\mathbf{P}||\mathbf{Q})=\frac{1}{2}\int \frac{{(p-q)}^2(p+q)}{pq}d\mu (\mathit{x})$$

(39)

2. Symmetrized KL divergence, called also J-divergence corresponding to Jeffreys entropy maximization [65,66]

$$\underset{\alpha \to 0}{lim}{D}_{AS1}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\underset{\alpha \to 1}{lim}{D}_{AS1}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=D_J(\mathbf{P}||\mathbf{Q})=\int (p-q)log\left(\frac{p}{q}\right)d\mu (\mathit{x})$$

(40)

An alternative wide class of symmetric divergences can be described by the following symmetric Alpha-divergence (Type-2):

$$\begin{array}{cccc}\hfill D_{AS2}^{(\alpha )}(\mathbf{P}||\mathbf{Q})& & \hfill =& D_A^{(\alpha )}\left(\mathbf{P}||\frac{\mathbf{P}+\mathbf{Q}}{2}\right)+D_A^{(\alpha )}\left(\mathbf{Q}||\frac{\mathbf{P}+\mathbf{Q}}{2}\right)\hfill \\ & & \hfill =& \frac{1}{\alpha (\alpha -1)}\int \left((p^{1-\alpha }+q^{1-\alpha }){\left(\frac{p+q}{2}\right)}^{\alpha }-(p+q)\right)d\mu (\mathit{x})\hfill \end{array}$$

The above measure admits the following prominent cases

• Triangular Discrimination [30,38]

  $$D_{AS2}^{(-1)}(\mathbf{P}||\mathbf{Q})=\frac{1}{2}{D}_{TD}(\mathbf{P}||\mathbf{Q})=\frac{1}{2}\int \frac{{(p-q)}^2}{p+q}d\mu (\mathit{x})$$

  (41)
• Symmetric Jensen–Shannon divergence [62,64]

  $$\underset{\alpha \to 0}{lim}{D}_{AS2}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=D_{JS}(\mathbf{P}||\mathbf{Q})=\int \left(plog\left(\frac{2p}{p+q}\right)+qlog\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x})$$

  (42)

  It is worth mentioning, that the Jensen–Shannon divergence is a symmetrized and smoothed variant of the Kullback–Leibler divergence, i.e., it can be interpreted as the average of the Kullback–Leibler divergences to the average distribution. For the normalized probability densities the Jensen–Shannon divergence is related to the Shannon entropy in the following sense:

  $$D_{JS}=H_S((\mathbf{P}+\mathbf{Q})/2)-(H_S(\mathbf{P})+H_S(\mathbf{Q}))/2$$

  (43)

  where $H_S(\mathbf{P})=-\int p(\mathit{x})logp(\mathit{x})d\mu (\mathit{x})$
• Arithmetic-Geometric divergence [39]

  $$\underset{\alpha \to 1}{lim}{D}_{AS2}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\int (p+q)log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x})$$

  (44)
• Symmetric Chi-square divergence [54]

  $$D_{AS2}^{(2)}(\mathbf{P}||\mathbf{Q})=\frac{1}{8}{D}_{{\chi }^2}(\mathbf{P}||\mathbf{Q})=\frac{1}{8}\int \frac{{(p-q)}^2(p+q)}{pq}d\mu (\mathit{x})$$

  (45)

The above Alpha-divergence is symmetric in its arguments $\mathbf{P}$ and $\mathbf{Q}$, and it is well-defined even if $\mathbf{P}$ and $\mathbf{Q}$ are not absolutely continuous. For example, for discrete $D_{AS2}^{(\alpha )}(\mathbf{P}||\mathbf{Q})$ is well-defined even if, for some indices $p_i$, it vanishes without vanishing $q_i$ or if $q_i$ vanishes without vanishing $p_i$ [54]. It is also lower- and upper-bounded, for example, the Jensen–Shannon divergence is bounded between 0 and 2 [36].

3. Family of Beta-Divergences

The basic Beta-divergence was introduced by Basu et al. [67] and Minami and Eguchi [15] and many researchers investigated their applications including [8,13,30,31,32,33,34,37,37,68,69,70,71,72], and references therein. The main motivation to investigate the beta divergence, at least from the practical point of view, is to develop highly robust in respect to outliers learning algorithms for clustering, feature extraction, classification and blind source separation. Until now the Beta-divergence has been successfully applied for robust PCA (Principal Component Analysis) and clustering [71], robust ICA (Independent Component Analysis) [15,68,69], and robust NMF/NTF [30,70,73,74,75,76].

First, let us define the basic asymmetric Beta-divergence between two unnormalized density functions by

$$D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=\int \left(p(\mathit{x})\frac{p^{\beta -1}(\mathit{x})-q^{\beta -1}(\mathit{x})}{\beta -1}-\frac{p^{\beta }(\mathit{x})-q^{\beta }(\mathit{x})}{\beta }\right)d\mu (\mathit{x}),\beta \in \mathbb{R}\setminus \{0,1\}$$

(46)

where β is a real number and, for $\beta =0,1$, is defined by continuity (see below for more explanation).

For discrete probability measures with mass functions $\mathbf{P}=[p_1,p_2,\dots ,p_n]$ and $\mathbf{Q}=[q_1,q_2,\dots ,q_n]$ the discrete Beta-divergence is defined as

$$D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=\sum _{i=1}^n{d}_B^{(\beta )}(p_i||q_i)=\sum _{i=1}^n\left(p_i\frac{p_i^{\beta -1}-q_i^{\beta -1}}{\beta -1}-\frac{p_i^{\beta }-q_i^{\beta }}{\beta }\right)\beta \in \mathbb{R}\setminus \{0,1\}$$

(47)

The Beta-divergence can be expressed via a generalized KL divergence as follows

$$D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=D_{GKL}^{({\beta }^{-1})}({\mathbf{P}}^{\beta }||{\mathbf{Q}}^{\beta })=-\frac{1}{\beta }\int \left(p^{\beta }{log}_{(\frac{1}{\beta })}\left(\frac{q^{\beta }}{p^{\beta }}\right)+p^{\beta }-q^{\beta }\right)d\mu (\mathit{x})\beta \in \mathbb{R}\setminus \{0,1\}$$

(48)

The above representation of the Beta-divergence indicates why it is robust to outliers for some values of the tuning parameter β and therefore, it is often better suited than others for some specific applications. For example, in sound processing, the speech power spectra can be modeled by exponential family densities of the form, whose for $\beta =0$ the Beta-divergence is no less than the Itakura–Saito distance (called also Itakura–Saito divergence or Itakura–Saito distortion measure or Burg cross entropy) [12,13,30,76,77,78,79]. In fact, the Beta-divergence has to be defined in limiting case for $\beta \to 0$ as the Itakura–Saito distance:

$$D_{IS}(\mathbf{P}||\mathbf{Q})=\underset{\beta \to 0}{lim}{D}_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=\int \left({\displaystyle log\frac{q}{p}+\frac{p}{q}-1}\right)d\mu (\mathit{x})$$

(49)

The Itakura and Saito distance was derived from the maximum likelihood (ML) estimation of speech spectra [77]. It was used as a measure of the distortion or goodness of fit between two spectra and is often used as a standard measure in the speech processing community due to the good perceptual properties of the reconstructed signals since it is scale invariant, Due to scale invariance low energy components of p have the same relative importance as high energy ones. This is especially important in the scenario in which the coefficients of p have a large dynamic range, such as in short-term audio spectra [30,76,79].

It is also interesting to note that, for $\beta =2$, we obtain the standard squared Euclidean ($L_2$-norm) distance, while for the singular case $\beta =1$, we obtain the KL I-divergence:

$$D_{KL}(\mathbf{P}||\mathbf{Q})=\underset{\beta \to 1}{lim}{D}_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=\int \left({\displaystyle plog\frac{p}{q}-p+q}\right)d\mu (\mathit{x})$$

(50)

Note, that we used here, the following formulas ${lim}_{\beta \to 0}\frac{p^{\beta }-q^{\beta }}{\beta }=log(p/q)$ and ${lim}_{\beta \to 0}\frac{p^{\beta }-1}{\beta }=logp$

Hence, the Beta-divergence can be represented in a more explicit form:

$$D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta (\beta -1)}\int \left(p^{\beta }(\mathit{x})+(\beta -1)q^{\beta }(\mathit{x})-\beta p(\mathit{x})q^{\beta -1}(\mathit{x})\right)d\mu (\mathit{x}),\beta \ne 0,1}\hfill \\ {\displaystyle \int \left({\displaystyle p(\mathit{x})log(\frac{p(\mathit{x})}{q(\mathit{x})})-p(\mathit{x})+q(\mathit{x})}\right)d\mu (\mathit{x}),\beta =1}\hfill \\ {\displaystyle \int \left({\displaystyle log(\frac{q(\mathit{x})}{p(\mathit{x})})+\frac{p(\mathit{x})}{q(\mathit{x})}-1}\right)d\mu (\mathit{x}),\beta =0}\hfill \end{array}\right.$$

(51)

We have shown that the basic Beta-divergence smoothly connects the Itakura–Saito distance and the squared Euclidean $L_2$-norm distance and passes through the KL I-divergence $D_{KL}(\mathbf{P}||\mathbf{Q})$. Such a parameterized connection is impossible in the family of the Alpha-divergences.

The choice of the tuning parameter β depends on the statistical distribution of data sets. For example, the optimal choice of the parameter β for the normal distribution is $\beta =2$, for the gamma distribution it is $\beta =0$, for the Poisson distribution $\beta =1$, and for the compound Poisson distribution $\beta \in (1,2)$ [15,31,32,33,34,68,69].

It is important to note that the Beta divergence can be derived from the Bregman divergence. The Bregman divergence is a pseudo-distance for measuring discrepancy between two values of density functions $p(\mathit{x})$ and $q(\mathit{x})$ [9,16,80]:

$$d_{\Phi }(p||q)=\Phi (p)-\Phi (q)-(p-q){\Phi }^{\prime }(q)$$

(52)

where $\Phi (t)$ is strictly convex real-valued function and ${\Phi }^{\prime }(q)$ is the derivative with respect to q. The total discrepancy between two functions $p(\mathit{x})$ and $q(\mathit{x})$ is given by

$$D_{\Phi }(\mathbf{P}||\mathbf{Q})=\int \left[\Phi (p(\mathit{x}))-\Phi (q(\mathit{x}))-(p(\mathit{x})-q(\mathit{x})){\Phi }^{\prime }(q(\mathit{x}))\right]d\mu (\mathit{x})$$

(53)

and it corresponds to the Φ-entropy of continuous probability measure $p(\mathit{x})\ge 0$ defined by

$$H_{\Phi }(\mathbf{P})=-\int \Phi (p(x))d\mu (\mathit{x})$$

(54)

Remark: The concept of divergence and entropy are closely related. Let ${\mathbf{Q}}_0$ be a uniform distribution for which

$${\mathbf{Q}}_0(\mathit{x})=const$$

(55)

(When $\mathit{x}$ is an infinite space ${\mathbb{R}}^n$ this might not be a probability distribution but is a measure). Then,

$$H(\mathbf{P})=-D(\mathbf{P}||{\mathbf{Q}}_0)+const$$

(56)

is regarded as the related entropy. This the negative of the divergence form $\mathbf{P}$ to the uniform distribution. On the other hand, given a concave entropy $H(\mathbf{P})$, we can define the related divergence as the Bregman divergence derived from a convex function $\Phi (\mathbf{P})=-H(\mathbf{P})$.

If $\mathit{x}$ takes discrete values on a certain space, the separable Bregman divergence is defined as $D_{\Phi }(\mathbf{P}||\mathbf{Q})={\sum }_{i=1}{d}_{\Phi }(p_i||q_i)={\sum }_{i=1}^n\left[\Phi (p_i)-\Phi (q_i)-(p_i-q_i){\Phi }^{\prime }(q_i))\right]$, where ${\Phi }^{\prime }(q)$ denotes derivative with respect to q. In a general (nonseparable) case for two vectors $\mathbf{P}$ and $\mathbf{Q}$, the Bregman divergence is defined as $D_{\Phi }(\mathbf{P}||\mathbf{Q})=\Phi (\mathbf{P})-\Phi (\mathbf{Q})-{(\mathbf{P}-\mathbf{Q})}^T\nabla \Phi (\mathbf{Q})$, where $\nabla \Phi (\mathbf{Q})$ is the gradient of Φ evaluated at $\mathbf{Q}$.

Note that $D_{\Phi }(\mathbf{P}||\mathbf{Q})$ equals the tail of the first-order Taylor expansion of $\Phi (\mathbf{P})$ at $\mathbf{Q}$. Bregman divergences include many prominent dissimilarity measures like the squared Euclidean distance, the Mahalanobis distance, the generalized Kullback–Leibler divergence and the Itakura–Saito distance.

It is easy to check that the Beta-divergence can be generated from the Bregman divergence using the following strictly convex continuous function [36,81]

$$\Phi (t)=\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta (\beta -1)}\left(t^{\beta }-\beta t+\beta -1\right),\beta \ne 0,1}\hfill \\ {\displaystyle tlog(t)-t+1,\beta =1}\hfill \\ {\displaystyle t-log(t)-1,\beta =0}\hfill \end{array}\right.$$

(57)

It is also interesting to note that the same generating function $f(u)=\Phi {(t)}_{|t=u}$ (with $\alpha =\beta$) can be used to generate the Alpha-divergence using the Csiszár–Morimoto f-divergence $D_f(\mathbf{P}||\mathbf{Q})=\int qf(p/q)d\mu (\mathit{x})$.

Furthermore, the Beta-divergence can be generated by a generalized f-divergence:

$${\displaystyle {\tilde{D}}_f(\mathbf{P}||\mathbf{Q})=\int q^{\beta }f(p/q)d\mu (\mathit{x})=\int q^{\beta }\tilde{f}\left(\frac{p^{\beta }}{q^{\beta }}\right)d\mu (\mathit{x})}$$

(58)

where $f(u)=\Phi {(t)}_{|t=u}$ with $u=p/q$ and

$$\tilde{f}(\tilde{u})=\frac{1}{1-\beta }\left({\tilde{u}}^{\frac{1}{\beta }}-\frac{1}{\beta }\tilde{u}+\frac{1}{\beta }-1\right)$$

(59)

is convex generating function with $\tilde{u}=p^{\beta }/q^{\beta }$.

The links between the Bregman and Beta-divergences are important, since the many well known fundamental properties of the Bregman divergence are also valid for the Beta-divergence [28,82]:

• Convexity: The Bregman divergence $D_{\Phi }(\mathbf{P}||\mathbf{Q})$ is always convex in the first argument $\mathbf{P}$, but is often not in the second argument $\mathbf{Q}$.
• Nonnegativity: The Bregman divergence is nonnegative $D_{\Phi }(\mathbf{P}||\mathbf{Q})\ge 0$ with zero $\mathbf{P}=\mathbf{Q}$.
• Linearity: Any positive linear combination of Bregman divergences is also a Bregman divergence, i.e.,

  $$D_{c_1{\Phi }_1+c_2{\Phi }_2}(\mathbf{P}||\mathbf{Q})=c_1{D}_{{\Phi }_1}(\mathbf{P}||\mathbf{Q})+c_2{D}_{{\Phi }_2}(\mathbf{P}||\mathbf{Q})$$

  (60)

  where $c_1,c_2$ are positive constants and ${\Phi }_1,{\Phi }_2$ are strictly convex functions.
• Invariance: The functional Bregman divergence is invariant under affine transforms $\Gamma (\mathbf{Q})=\Phi (\mathbf{Q})+\int a(x-x^{\prime })q(x^{\prime })d{x}^{\prime }+c$ for positive measures $\mathbf{P}$ and $\mathbf{Q}$ to linear and arbitrary constant terms [28,82], i.e.,

  $$D_{\Gamma }(\mathbf{P}||\mathbf{Q})=D_{\Phi }(\mathbf{P}||\mathbf{Q})$$

  (61)
• The three-point property generalizes the “Law of Cosines”:

  $$D_{\Phi }(\mathbf{P}||\mathbf{Q})=D_{\Phi }(\mathbf{P}||\mathbf{Z})+D_{\Phi }(\mathbf{Z}||\mathbf{Q})-{(\mathbf{P}-\mathbf{Z})}^T(\frac{\delta }{\delta \mathbf{Q}}\Phi (\mathbf{Q})-\frac{\delta }{\delta \mathbf{Z}}\Phi (\mathbf{Z}))$$

  (62)
• Generalized Pythagoras Theorem:

  $$D_{\Phi }(\mathbf{P}||\mathbf{Q})\ge D_{\Phi }(\mathbf{P}||P_{\Omega }(\mathbf{Q}))+D_{\Phi }(P_{\Omega }(\mathbf{Q})||\mathbf{Q})$$

  (63)

  where ${\displaystyle P_{\Omega }(\mathbf{Q})=\underset{\mathbf{\omega }\in \Omega }{argmin}{D}_{\Phi }(\mathbf{\omega }||\mathbf{Q})}$ is the Bregman projection onto the convex set Ω and $\mathbf{P}\in \Omega$. When Ω is an affine set then it holds with equality. This is proved to be the generalized Pythagorean relation in terms of information geometry.

For the Beta-divergence (46) the first and second-order Fréchet derivative with respect to $\mathbf{Q}$ are given by [28,76]

$$\frac{\delta D_B^{(\beta )}}{\delta q}=q^{\beta -2}(q-p),\frac{{\delta }^2{D}_B^{(\beta )}}{\delta q^2}=q^{\beta -3}((\beta -1)q-(\beta -2)p)$$

(64)

Hence, we conclude that the Beta-divergence has a single global minimum equal to zero for $\mathbf{P}=\mathbf{Q}$ and increases with $|p-q|$. Moreover, the Beta divergence is strictly convex for $q(\mathit{x})>0$ only for $\beta \in [1,2]$. For $\beta =0$ (Itakura–Saito distance), it is convex if $\int q^{-3}(2p-q)d\mu (\mathit{x})\ge 0$ i.e., if $q/p\le 2$ [78].

3.1. Generation of Family of Beta-divergences Directly from Family of Alpha-Divergences

It should be noted that in the original works [15,67,68,69] they considered only the Beta-divergence function for $\beta \ge 1$. Moreover, they did not consider the whole range of non-positive values for parameter β, especially $\beta =0$, for which we have the important Itakura–Saito distance. Furthermore, similar to the Alpha-divergences there exist an associated family of Beta-divergences and as special cases a family of generalized Itakura–Saito like distances. The fundamental question arises: How to generate a whole family of Beta-divergences or what is the relationships or correspondences between the Alpha- and Beta-divergences. In fact, on the basis of our considerations above, it is easy to find that the complete set of Beta-divergences can be obtained from the Alpha-divergences and conversely the Alpha-divergences, can obtained directly from Beta-divergences.

In order to obtain a Beta-divergence from the corresponding (associated) Alpha-divergence, we need to apply the following nonlinear transformations:

$$p\to p^{\beta },q\to q^{\beta }\mathrm{and}\alpha ={\beta }^{-1}$$

(65)

For example, using these transformations (substitutions) for a basic asymmetric Alpha-divergence (4) and assuming that $\alpha ={(\beta )}^{-1}$, we obtain the following divergence

$$D_A^{(\beta )}(\mathbf{P}||\mathbf{Q})={\beta }^2\int \left(\frac{-\beta p{q}^{\beta -1}+p^{\beta }+(\beta -1)q^{\beta }}{\beta (\beta -1)}\right)d\mu (\mathit{x})$$

(66)

Observe that, by ignoring the scaling factor ${\beta }^2$, we obtain the basic asymmetric Beta-divergence defined by Equation (46).

In fact, there exists the same link between the whole family of Alpha-divergences and the family of Beta-divergences (see Table 2).

For example, we can derive a symmetric Beta-divergence from the symmetric Alpha-divergence (Type-1) (38):

$$\begin{array}{ccc}\hfill D_{BS1}^{(\beta )}(\mathbf{P}||\mathbf{Q})& =\hfill & \hfill D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})+D_B^{(\beta )}(\mathbf{Q}||\mathbf{P})\\ & =\hfill & \hfill \frac{1}{\beta -1}\int \left((p-q)(p^{\beta -1}-q^{\beta -1})\right)d\mu (\mathit{x})\end{array}$$

It is interesting to note that, in special cases, we obtain:

Symmetric KL of J-divergence [65]:

$$D_{BS1}^{(1)}=\underset{\beta \to 1}{lim}{D}_{BS1}^{(\beta )}=\int (p-q)log\left(\frac{p}{q}\right)d\mu (\mathit{x})$$

(67)

and symmetric Chi-square divergence [54]

$$D_{BS1}^{(0)}=\underset{\beta \to 0}{lim}{D}_{BS1}^{(\beta )}=\int \frac{{(p-q)}^2}{pq}d\mu (\mathit{x})$$

(68)

Analogously, from the symmetric Alpha-divergence (Type-2), we obtain

$$D_{BS2}^{(\beta )}(\mathbf{P}||\mathbf{Q})=\frac{1}{\beta -1}\int \left(p^{\beta }+q^{\beta }-(p^{\beta -1}+q^{\beta -1}){\left(\frac{p^{\beta }+q^{\beta }}{2}\right)}^{\frac{1}{\beta }}\right)d\mu (\mathit{x})$$

(69)

Table 2. Family of Alpha-divergences and corresponding Beta-divergences. We applied the following transformations $p\to p^{\beta },q\to q^{\beta },\alpha =1/\beta$. Note that $D_A^{(1)}(\mathbf{P}||\mathbf{Q})=D_B^{(1)}(\mathbf{P}||\mathbf{Q})$ and they represents for $\alpha =\beta =1$ extended family of KL divergences. Furthermore, Beta-divergences for $\beta =0$ describe the family of generalized (extended) Itakura–Saito like distances.

Alpha-divergence ${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})}$	Beta-divergence ${\displaystyle D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})}$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{q}{p}\right)+p-q\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta (\beta -1)}\int \left(p^{\beta }+(\beta -1)q^{\beta }-\beta p{q}^{\beta -1}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(log\left(\frac{q}{p}\right)+\frac{p}{q}-1\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left({\displaystyle {(\frac{p+q}{2})}^{\alpha }{q}^{1-\alpha }-\frac{\alpha }{2}p+(\frac{\alpha }{2}-1)q}\right)d\mu (\mathit{x})}}{\alpha (\alpha -1)},}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{2q}{p+q}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p+q}{2q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left({\displaystyle p^{\beta }+(2\beta -1)q^{\beta }-2\beta q^{\beta -1}{\left(\frac{p^{\beta }+q^{\beta }}{2}\right)}^{\frac{1}{\beta }}}\right)d\mu (\mathit{x})}}{\beta (\beta -1)},}\hfill \\ {\displaystyle \int \left(log(\frac{q}{p})+2(\sqrt{\frac{p}{q}}-1)\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p_1+q}{2q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha -1}\int (p-q)\left[{\left(\frac{p+q}{2q}\right)}^{\alpha -1}-1\right]d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \frac{{(p-q)}^2}{p+q}d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta (\beta -1)}\int (p^{\beta }-q^{\beta })\left(1-\frac{2^{\frac{\beta -1}{\beta }}{q}^{\beta -1}}{{\left(p^{\beta }+q^{\beta }\right)}^{\frac{\beta -1}{\beta }}}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(\sqrt{\frac{p}{q}}-1\right)log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }{q}^{1-\alpha }+p^{1-\alpha }{q}^{\alpha }-p-q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\alpha =0,1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta -1}\int \left(p^{\beta }+q^{\beta }-p{q}^{\beta -1}-p^{\beta -1}q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \frac{{(p-q)}^2}{pq}d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle {\displaystyle \frac{1}{1-\alpha }\int \left({\displaystyle \frac{p+q}{2}-{\left(\frac{p^{\alpha }+q^{\alpha }}{2}\right)}^{\frac{1}{\alpha }}}\right)d\mu (\mathit{x}),}}\hfill \\ {\displaystyle \int \left(\frac{\sqrt{p}-\sqrt{q}{)}^2}{2}\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle H_S\left(\frac{\mathbf{P}+\mathbf{Q}}{2}\right)-\frac{H_S(\mathbf{P})+H_S(\mathbf{Q})}{2},\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta (\beta -1)}\int \left({\displaystyle \frac{p^{\beta }+q^{\beta }}{2}-{\left({\displaystyle \frac{p+q}{2}}\right)}^{\beta }}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle H_S\left(\frac{\mathbf{P}+\mathbf{Q}}{2}\right)-\frac{H_S(\mathbf{P})+H_S(\mathbf{Q})}{2},\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left((p^{1-\alpha }+q^{1-\alpha }){\left({\displaystyle \frac{p+q}{2}}\right)}^{\alpha }-p-q\right)d\mu (\mathit{x})}}{\alpha (\alpha -1)},}\hfill \\ {\displaystyle \int \left(plog\left(\frac{2p}{p+q}\right)+qlog\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int (p+q)log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle {\displaystyle \frac{1}{\beta -1}\int \left(p^{\beta }+q^{\beta }-(p^{\beta -1}+q^{\beta -1}){\left(\frac{p^{\beta }+q^{\beta }}{2}\right)}^{\frac{1}{\beta }}\right)d\mu (\mathit{x}),}}\hfill \\ {\displaystyle \int \left(\sqrt{\frac{p}{q}}+\sqrt{\frac{q}{p}}-2\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int (p+q)log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$

Table 2. Family of Alpha-divergences and corresponding Beta-divergences. We applied the following transformations $p\to p^{\beta },q\to q^{\beta },\alpha =1/\beta$. Note that $D_A^{(1)}(\mathbf{P}||\mathbf{Q})=D_B^{(1)}(\mathbf{P}||\mathbf{Q})$ and they represents for $\alpha =\beta =1$ extended family of KL divergences. Furthermore, Beta-divergences for $\beta =0$ describe the family of generalized (extended) Itakura–Saito like distances.

Alpha-divergence ${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})}$	Beta-divergence ${\displaystyle D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})}$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{q}{p}\right)+p-q\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta (\beta -1)}\int \left(p^{\beta }+(\beta -1)q^{\beta }-\beta p{q}^{\beta -1}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(log\left(\frac{q}{p}\right)+\frac{p}{q}-1\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left({\displaystyle {(\frac{p+q}{2})}^{\alpha }{q}^{1-\alpha }-\frac{\alpha }{2}p+(\frac{\alpha }{2}-1)q}\right)d\mu (\mathit{x})}}{\alpha (\alpha -1)},}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{2q}{p+q}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p+q}{2q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left({\displaystyle p^{\beta }+(2\beta -1)q^{\beta }-2\beta q^{\beta -1}{\left(\frac{p^{\beta }+q^{\beta }}{2}\right)}^{\frac{1}{\beta }}}\right)d\mu (\mathit{x})}}{\beta (\beta -1)},}\hfill \\ {\displaystyle \int \left(log(\frac{q}{p})+2(\sqrt{\frac{p}{q}}-1)\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p_1+q}{2q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha -1}\int (p-q)\left[{\left(\frac{p+q}{2q}\right)}^{\alpha -1}-1\right]d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \frac{{(p-q)}^2}{p+q}d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta (\beta -1)}\int (p^{\beta }-q^{\beta })\left(1-\frac{2^{\frac{\beta -1}{\beta }}{q}^{\beta -1}}{{\left(p^{\beta }+q^{\beta }\right)}^{\frac{\beta -1}{\beta }}}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(\sqrt{\frac{p}{q}}-1\right)log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }{q}^{1-\alpha }+p^{1-\alpha }{q}^{\alpha }-p-q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\alpha =0,1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta -1}\int \left(p^{\beta }+q^{\beta }-p{q}^{\beta -1}-p^{\beta -1}q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \frac{{(p-q)}^2}{pq}d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle {\displaystyle \frac{1}{1-\alpha }\int \left({\displaystyle \frac{p+q}{2}-{\left(\frac{p^{\alpha }+q^{\alpha }}{2}\right)}^{\frac{1}{\alpha }}}\right)d\mu (\mathit{x}),}}\hfill \\ {\displaystyle \int \left(\frac{\sqrt{p}-\sqrt{q}{)}^2}{2}\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle H_S\left(\frac{\mathbf{P}+\mathbf{Q}}{2}\right)-\frac{H_S(\mathbf{P})+H_S(\mathbf{Q})}{2},\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\beta (\beta -1)}\int \left({\displaystyle \frac{p^{\beta }+q^{\beta }}{2}-{\left({\displaystyle \frac{p+q}{2}}\right)}^{\beta }}\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle H_S\left(\frac{\mathbf{P}+\mathbf{Q}}{2}\right)-\frac{H_S(\mathbf{P})+H_S(\mathbf{Q})}{2},\beta =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left((p^{1-\alpha }+q^{1-\alpha }){\left({\displaystyle \frac{p+q}{2}}\right)}^{\alpha }-p-q\right)d\mu (\mathit{x})}}{\alpha (\alpha -1)},}\hfill \\ {\displaystyle \int \left(plog\left(\frac{2p}{p+q}\right)+qlog\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int (p+q)log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle {\displaystyle \frac{1}{\beta -1}\int \left(p^{\beta }+q^{\beta }-(p^{\beta -1}+q^{\beta -1}){\left(\frac{p^{\beta }+q^{\beta }}{2}\right)}^{\frac{1}{\beta }}\right)d\mu (\mathit{x}),}}\hfill \\ {\displaystyle \int \left(\sqrt{\frac{p}{q}}+\sqrt{\frac{q}{p}}-2\right)d\mu (\mathit{x}),\beta =0}\hfill \\ {\displaystyle \int (p+q)log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\beta =1}\hfill \end{array}\right.$

It should be noted that in special cases, we obtain:

The Arithmetic-Geometric divergence [38,39]:

$$D_{BS2}^{(1)}=\underset{\beta \to 1}{lim}{D}_{BS2}^{(\beta )}=\int (p+q)log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),$$

(70)

and a symmetrized Itakura–Saito distance (called also the COSH distance) [12,13]:

$$D_{BS2}^{(0)}=\underset{\beta \to 0}{lim}{D}_{BS2}^{(\beta )}=\int \left(\sqrt{\frac{p}{q}}+\sqrt{\frac{q}{p}}-2\right)d\mu (\mathit{x})=\int \frac{{(\sqrt{p}-\sqrt{q})}^2}{\sqrt{pq}}d\mu (\mathit{x})$$

(71)

4. Family of Gamma-Divergences Generated from Beta- and Alpha-Divergences

A basic asymmetric Gamma-divergence has been proposed very recently by Fujisawa and Eguchi [35] as a very robust similarity measure with respect to outliers:

$$D_G^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\frac{1}{\gamma (\gamma -1)}log\left(\frac{\left({\displaystyle \int p^{\gamma }(\mathit{x})d\mu (\mathit{x})}\right){\left({\displaystyle \int q^{\gamma }(\mathit{x})d\mu (\mathit{x})}\right)}^{\gamma -1}}{{\left({\displaystyle \int p(\mathit{x})q^{\gamma -1}(\mathit{x})d\mu (\mathit{x})}\right)}^{\gamma }}\right)$$

(72)

The Gamma-divergence employs the nonlinear transformation (log) for cumulative patterns and the terms $p,q$ are not separable. The main motivation for employing the Gamma divergence is that it allows “super” robust estimation of some parameters in presence of outlier. In fact, the authors demonstrated that the bias caused by outliers can become sufficiently small even in the case of very heavy contamination and that some contamination can be naturally and automatically neglected [35,37].

In this paper, we show that we can formulate the whole family of Gamma-divergences generated directly from Alpha- and also Beta-divergences. In order to obtain a robust Gamma-divergence from an Alpha- or Beta-divergence, we use the following transformations (see also Table 3):

$$c_0\int p^{c_1}(\mathit{x})q^{c_2}(\mathit{x})d\mu (\mathit{x})\to log{\left(\int p^{c_1}(\mathit{x})q^{c_2}(\mathit{x})d\mu (\mathit{x})\right)}^{c_0}$$

(73)

where $c_0,c_1$ and $c_2$ are real constants and $\gamma =\alpha$.

Applying the above transformation to all monomials to the basic Alpha-divergence (4), we obtain a new divergence referred to as here the Alpha-Gamma-divergence:

$$\begin{array}{cccc}\hfill D_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})& & \hfill =& log{\left(\int p^{\gamma }(\mathit{x})q^{1-\gamma }(\mathit{x})d\mu (\mathit{x})\right)}^{\frac{1}{\gamma (\gamma -1)}}-log{\left(\int p(\mathit{x})d\mu (\mathit{x})\right)}^{\frac{1}{\gamma -1}}+log{\left(\int q(\mathit{x})d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }}\hfill \\ & & \hfill =& {\displaystyle \frac{1}{\gamma (\gamma -1)}log\left(\frac{\int p^{\gamma }(\mathit{x})q^{1-\gamma }(\mathit{x})d\mu (\mathit{x})}{{\left(\int p(\mathit{x})d\mu (\mathit{x})\right)}^{\gamma }{\left(\int q(\mathit{x})d\mu (\mathit{x})\right)}^{1-\gamma }}\right)}\hfill \end{array}$$

The asymmetric Alpha-Gamma-divergence has the following important properties:

• $D_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})\ge 0$. The equality holds if and only if $\mathbf{P}=c\mathbf{Q}$ for a positive constant c.
• It is scale invariant for any value of γ, that is, $D_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{AG}^{(\gamma )}(c_1\mathbf{P}||c_2\mathbf{Q})$, for arbitrary positive scaling constants $c_1,c_2$.
• The Alpha-Gamma divergence is equivalent to the normalized Alpha-Rényi divergence (25), i.e.,

  $$\begin{array}{cccc}\hfill D_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})& & \hfill =& {\displaystyle \frac{1}{\gamma (\gamma -1)}log\left(\frac{{\displaystyle \int p^{\gamma }(\mathit{x})q^{1-\gamma }(\mathit{x})d\mu (\mathit{x})}}{{\left({\displaystyle \int p(\mathit{x})d\mu (\mathit{x})}\right)}^{\gamma }{\left({\displaystyle \int q(\mathit{x})d\mu (\mathit{x})}\right)}^{1-\gamma }}\right)}\hfill \\ & & \hfill =& {\displaystyle \frac{1}{\gamma (\gamma -1)}log\left(\int {\overline{p}}^{\gamma }(\mathit{x}){\overline{q}}^{1-\gamma }(\mathit{x})d\mu (\mathit{x})\right)}\hfill \\ & & \hfill =& {\displaystyle \frac{1}{\gamma -1}log{\left(\int \overline{q}(\mathit{x}){\left(\frac{\overline{p}(\mathit{x})}{\overline{q}(\mathit{x})}\right)}^{\gamma }d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }}=D_{AR}^{(\gamma )}(\overline{\mathbf{P}}||\overline{\mathbf{Q}})}\hfill \end{array}$$

  for $\alpha =\gamma$ and normalized densities $\overline{p}(\mathit{x})=p(\mathit{x})/\int (p(\mathit{x})d\mu (\mathit{x})$ and $\overline{q}(\mathit{x})=q(\mathit{x})/\int (q(\mathit{x})d\mu (\mathit{x})$.
• It can be expressed via generalized weighted mean:

  $$D_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\frac{1}{\gamma -1}log\left({\overline{M}}_{\gamma }\left\{\overline{q};\frac{\overline{p}}{\overline{q}}\right\}\right)$$

  (74)

  where the weighted mean is defined as ${\displaystyle {\overline{M}}_{\gamma }\left\{{\displaystyle \overline{q};\frac{\overline{p}}{\overline{q}}}\right\}={\left(\int \overline{q}(\mathit{x}){\left({\displaystyle \frac{\overline{p}(\mathit{x})}{\overline{q}(\mathit{x})}}\right)}^{\gamma }d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }}}$.
• As $\gamma \to 0$, the Alpha-Gamma-divergence becomes the Kullback–Leibler divergence:

  $${\displaystyle \underset{\gamma \to 0}{lim}{D}_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{KL}(\overline{\mathbf{P}}||\overline{\mathbf{Q}})=\int \overline{p}(\mathit{x})log\frac{\overline{p}(\mathit{x})}{\overline{q}(\mathit{x})}d\mu (\mathit{x})}$$

  (75)
• For $\gamma \to 1$, the Alpha-Gamma-divergence can be expressed by the reverse Kullback–Leibler divergence:

  $${\displaystyle \underset{\gamma \to 1}{lim}{D}_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{KL}(\overline{\mathbf{Q}}||\overline{\mathbf{P}})=\int \overline{q}(\mathit{x})log\frac{\overline{q}(\mathit{x})}{\overline{p}(\mathit{x})}d\mu (\mathit{x})}$$

  (76)

In a similar way, we can generate the whole family of Alpha-Gamma-divergences from the family of Alpha-divergences, which are summarized in Table 3.

It is interesting to note that using the above transformations (73) with $\gamma =\beta$, we can generate another family of Gamma divergences, referred to as Beta-Gamma divergences.

In particular, using the nonlinear transformations (73) for the basic asymmetric Beta-divergence (46), we obtain the Gamma-divergence (72) [35] referred to as here a Beta-Gamma-divergence ($D_G^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})$)

$$\begin{array}{cccc}\hfill D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})& & \hfill =& \frac{1}{\gamma (\gamma -1)}\left[log(\int p^{\gamma }d\mu (\mathit{x}))+(\gamma -1)log(\int q^{\gamma }d\mu (\mathit{x}))-\gamma log(\int p{q}^{\gamma -1}d\mu (\mathit{x}))\right]\hfill \\ & & \hfill =& log\left(\frac{{\left({\displaystyle \int p{q}^{\gamma -1}d\mu (\mathit{x})}\right)}^{\frac{1}{1-\gamma }}}{{\left({\displaystyle \int p^{\gamma }d\mu (\mathit{x})}\right)}^{\frac{1}{\gamma (1-\gamma )}}{\left({\displaystyle \int q^{\gamma }d\mu (\mathit{x})}\right)}^{\frac{-1}{\gamma }}}\right)\hfill \\ & & \hfill =& \frac{1}{1-\gamma }log\left({\displaystyle \int {\tilde{q}}^{\gamma }(\mathit{x})\left(\frac{\tilde{p}(\mathit{x})}{\tilde{q}(\mathit{x})}\right)d\mu (\mathit{x})}\right)\hfill \end{array}$$

where

$$\tilde{p}(\mathit{x})=\frac{p(\mathit{x})}{{\left(\int p^{\gamma }(\mathit{x})d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }}},\tilde{q}(\mathit{x})=\frac{q(\mathit{x})}{{\left(\int q^{\gamma }(\mathit{x})d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }}}$$

(77)

Analogously, for discrete densities we can express the Beta-Gamma-divergence via generalized power means also known as the power mean or Hölder means as follows

$$D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=-log\frac{{\left({\displaystyle \sum _{i=1}^n{p}_i{q}_i^{\gamma -1}}\right)}^{\frac{1}{\gamma -1}}}{{\left({\displaystyle \sum _{i=1}^n{p}_i^{\gamma }}\right)}^{\frac{1}{\gamma (\gamma -1)}}{\left({\displaystyle \sum _{i=1}^n{q}_i^{\gamma }}\right)}^{\frac{1}{\gamma }}}$$

(78)

Hence,

$$D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=-log\frac{{\left({\displaystyle \sum _{i=1}^n\frac{p_i}{q_i}{q}_i^{\gamma }}\right)}^{\frac{1}{\gamma -1}}}{{\left[{\displaystyle {\left(\sum _{i=1}^n{p}_i^{\gamma }\right)}^{\frac{1}{\gamma }}}\right]}^{\frac{1}{\gamma -1}}{\left({\displaystyle \sum _{i=1}^n{q}_i^{\gamma }}\right)}^{\frac{1}{\gamma }}}=-log{\left(\frac{{\displaystyle \frac{1}{n}\sum _{i=1}^n\frac{p_i}{q_i}{q}_i^{\gamma }}}{{\left({\displaystyle \frac{1}{n}\sum _{i=1}^n{p}_i^{\gamma }}\right)}^{\frac{1}{\gamma }}{\left[{\left({\displaystyle \frac{1}{n}\sum _{i=1}^n{q}_i^{\gamma }}\right)}^{\frac{1}{\gamma }}\right]}^{\gamma -1}}\right)}^{\frac{1}{\gamma -1}}$$

and finally

$$D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\frac{1}{1-\gamma }log\frac{{\displaystyle \frac{1}{n}\sum _{i=1}^n\frac{p_i}{q_i}{q}_i^{\gamma }}}{{\displaystyle M_{\gamma }\{p_i\}{\left[M_{\gamma }\{q_i\}\right]}^{\gamma -1}}}$$

(79)

where the (generalized) power mean of the order-γ is defined as

$$M_{\gamma }\{p_i\}={\left(\frac{1}{n}\sum _{i=1}^n{p}_i^{\gamma }\right)}^{\frac{1}{\gamma }}$$

(80)

In the special cases, we obtain standard harmonic mean ($\gamma =-1)$, geometric mean ($\gamma =0$), arithmetic mean($\gamma =1$), and squared root mean $\gamma =2$ with the following relations:

$$M_{-\infty }\{p_i\}\le M_{-1}(\{p_i\})\le M_0\{p_i\}\le M_1\{p_i\}\le M_2\{p_i\}\le M_{\infty }\{p_i\},$$

(81)

with $M_0\{p_i\}={lim}_{\gamma \to 0}{M}_{\gamma }\{p_i\}={\left({\prod }_{i=1}^n{p}_i\right)}^{1/n}$, $M_{-\infty }\{p_i\}={min}_i\{p_i\}$ and $M_{\infty }\{p_i\}={max}_i\{p_i\}$.

Table 3. Family of Alpha-divergences and corresponding robust Alpha-Gamma-divergences; $\overline{p}(\mathit{x})=p(\mathit{x})/(\int p(\mathit{x})d\mu (\mathit{x})),\overline{q}(\mathit{x})=q(\mathit{x})/(\int q(\mathit{x})d\mu (\mathit{x}))$. For $\gamma =0,1$, we obtained a generalized robust KL divergences. Note that Gamma divergences are expressed compactly via generalized power means. (see also Table 4 for more direct representations).

Alpha-divergence ${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})}$	Robust Alpha-Gamma-divergence ${\displaystyle D_{AG}^{(\gamma )}(c\mathbf{P}||c\mathbf{Q})}$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{q}{p}\right)+p-q\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\gamma -1}log{\left(\int \overline{q}{\left(\frac{\overline{p}}{\overline{q}}\right)}^{\gamma }d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }},}\hfill \\ {\displaystyle \int \overline{q}log\left(\frac{\overline{q}}{\overline{p}}\right)d\mu (\mathit{x}),\gamma =0}\hfill \\ {\displaystyle \int \overline{p}log\left(\frac{\overline{p}}{\overline{q}}\right)d\mu (\mathit{x}),\gamma =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{1-\alpha }{q}^{\alpha }+(\alpha -1)p-\alpha q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{q}{p}\right)+p-q\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\gamma -1}log{\left(\int \overline{p}{\left(\frac{\overline{q}}{\overline{p}}\right)}^{\gamma }d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }},}\hfill \\ {\displaystyle \int \overline{p}log\left(\frac{\overline{p}}{\overline{q}}\right)d\mu (\mathit{x}),\gamma =0}\hfill \\ {\displaystyle \int \overline{q}log\left(\frac{\overline{q}}{\overline{p}}\right)d\mu (\mathit{x}),\gamma =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left({\left(\frac{p+q}{2}\right)}^{\alpha }{q}^{1-\alpha }-\frac{\alpha }{2}p-(1-\frac{\alpha }{2})q\right)d\mu (\mathit{x})}}{\alpha (\alpha -1)},}\hfill \\ {\displaystyle \int \left(qlog\left({\displaystyle \frac{2q}{p+q}}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p+q}{2q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\gamma -1}log\left[{\displaystyle {\left(\frac{\int qd\mu (\mathit{x})}{\int pd\mu (\mathit{x})}\right)}^{\frac{1}{2}}{\left({\displaystyle \int \overline{q}{\left(\frac{p+q}{2q}\right)}^{\gamma }d\mu (\mathit{x})}\right)}^{\frac{1}{\gamma }}}\right]}\hfill \\ {\displaystyle \int \left((\overline{q}log\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x}),\gamma =0}\hfill \\ {\displaystyle \int \left(\frac{\overline{p}+\overline{q}}{2}log\left(\frac{p+q}{2q}\right)\right)d\mu (\mathit{x}),\gamma =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha -1}\int \left[{\left(\frac{p+q}{2q}\right)}^{\alpha -1}-1\right](p-q)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle log{\left(\frac{{\displaystyle \int (\overline{p}{\left(\frac{p+q}{2q}\right)}^{\gamma -1})d\mu (\mathit{x})}}{{\displaystyle \int (\overline{q}{\left(\frac{p+q}{2q}\right)}^{\gamma -1})d\mu (\mathit{x})}}\right)}^{\frac{1}{\gamma -1}}}\hfill \\ {\displaystyle \int (\overline{p}-\overline{q})log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}),\gamma =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }{q}^{1-\alpha }+p^{1-\alpha }{q}^{\alpha }-p-q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\alpha =0,1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\gamma -1}log{\left[\int \overline{q}{\left(\frac{p}{q}\right)}^{\gamma }d\mu (\mathit{x})\int \overline{p}{\left(\frac{q}{p}\right)}^{\gamma }d\mu (\mathit{x})\right]}^{\frac{1}{\gamma }}}\hfill \\ {\displaystyle \int (\overline{p}-\overline{q})log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\gamma =0,1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left((p^{1-\alpha }+q^{1-\alpha }){\left({\displaystyle \frac{p+q}{2}}\right)}^{\alpha }-p-q\right)d\mu (\mathit{x})}}{\alpha (\alpha -1)},}\hfill \\ {\displaystyle \int \left(plog\left(\frac{2p}{p+q}\right)+qlog\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int (p+q)log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}\frac{1}{\gamma -1}log{\left[{\displaystyle \int \overline{p}{\left({\displaystyle \frac{p+q}{2p}}\right)}^{\gamma }d\mu (\mathit{x})\int \overline{q}{\left({\displaystyle \frac{p+q}{2q}}\right)}^{\gamma }d\mu (\mathit{x})}\right]}^{\frac{1}{\gamma }}\hfill \\ {\displaystyle \int \left(\overline{p}log\left(\frac{2p}{p+q}\right)+\overline{q}log\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x}),\gamma =0}\hfill \\ {\displaystyle \int (\overline{p}+\overline{q})log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\gamma =1.}\hfill \end{array}\right.$

Table 3. Family of Alpha-divergences and corresponding robust Alpha-Gamma-divergences; $\overline{p}(\mathit{x})=p(\mathit{x})/(\int p(\mathit{x})d\mu (\mathit{x})),\overline{q}(\mathit{x})=q(\mathit{x})/(\int q(\mathit{x})d\mu (\mathit{x}))$. For $\gamma =0,1$, we obtained a generalized robust KL divergences. Note that Gamma divergences are expressed compactly via generalized power means. (see also Table 4 for more direct representations).

Alpha-divergence ${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})}$	Robust Alpha-Gamma-divergence ${\displaystyle D_{AG}^{(\gamma )}(c\mathbf{P}||c\mathbf{Q})}$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{q}{p}\right)+p-q\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\gamma -1}log{\left(\int \overline{q}{\left(\frac{\overline{p}}{\overline{q}}\right)}^{\gamma }d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }},}\hfill \\ {\displaystyle \int \overline{q}log\left(\frac{\overline{q}}{\overline{p}}\right)d\mu (\mathit{x}),\gamma =0}\hfill \\ {\displaystyle \int \overline{p}log\left(\frac{\overline{p}}{\overline{q}}\right)d\mu (\mathit{x}),\gamma =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{1-\alpha }{q}^{\alpha }+(\alpha -1)p-\alpha q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int \left(plog\left(\frac{p}{q}\right)-p+q\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(qlog\left(\frac{q}{p}\right)+p-q\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\gamma -1}log{\left(\int \overline{p}{\left(\frac{\overline{q}}{\overline{p}}\right)}^{\gamma }d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }},}\hfill \\ {\displaystyle \int \overline{p}log\left(\frac{\overline{p}}{\overline{q}}\right)d\mu (\mathit{x}),\gamma =0}\hfill \\ {\displaystyle \int \overline{q}log\left(\frac{\overline{q}}{\overline{p}}\right)d\mu (\mathit{x}),\gamma =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left({\left(\frac{p+q}{2}\right)}^{\alpha }{q}^{1-\alpha }-\frac{\alpha }{2}p-(1-\frac{\alpha }{2})q\right)d\mu (\mathit{x})}}{\alpha (\alpha -1)},}\hfill \\ {\displaystyle \int \left(qlog\left({\displaystyle \frac{2q}{p+q}}\right)+\frac{p-q}{2}\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int \left(\frac{p+q}{2}log\left(\frac{p+q}{2q}\right)-\frac{p-q}{2}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\gamma -1}log\left[{\displaystyle {\left(\frac{\int qd\mu (\mathit{x})}{\int pd\mu (\mathit{x})}\right)}^{\frac{1}{2}}{\left({\displaystyle \int \overline{q}{\left(\frac{p+q}{2q}\right)}^{\gamma }d\mu (\mathit{x})}\right)}^{\frac{1}{\gamma }}}\right]}\hfill \\ {\displaystyle \int \left((\overline{q}log\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x}),\gamma =0}\hfill \\ {\displaystyle \int \left(\frac{\overline{p}+\overline{q}}{2}log\left(\frac{p+q}{2q}\right)\right)d\mu (\mathit{x}),\gamma =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha -1}\int \left[{\left(\frac{p+q}{2q}\right)}^{\alpha -1}-1\right](p-q)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle log{\left(\frac{{\displaystyle \int (\overline{p}{\left(\frac{p+q}{2q}\right)}^{\gamma -1})d\mu (\mathit{x})}}{{\displaystyle \int (\overline{q}{\left(\frac{p+q}{2q}\right)}^{\gamma -1})d\mu (\mathit{x})}}\right)}^{\frac{1}{\gamma -1}}}\hfill \\ {\displaystyle \int (\overline{p}-\overline{q})log\left(\frac{p+q}{2q}\right)d\mu (\mathit{x}),\gamma =1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha (\alpha -1)}\int \left(p^{\alpha }{q}^{1-\alpha }+p^{1-\alpha }{q}^{\alpha }-p-q\right)d\mu (\mathit{x}),}\hfill \\ {\displaystyle \int (p-q)log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\alpha =0,1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\displaystyle \frac{1}{\gamma -1}log{\left[\int \overline{q}{\left(\frac{p}{q}\right)}^{\gamma }d\mu (\mathit{x})\int \overline{p}{\left(\frac{q}{p}\right)}^{\gamma }d\mu (\mathit{x})\right]}^{\frac{1}{\gamma }}}\hfill \\ {\displaystyle \int (\overline{p}-\overline{q})log\left(\frac{p}{q}\right)d\mu (\mathit{x}),\gamma =0,1}\hfill \end{array}\right.$
$\left\{\begin{array}{c}{\displaystyle \frac{{\displaystyle \int \left((p^{1-\alpha }+q^{1-\alpha }){\left({\displaystyle \frac{p+q}{2}}\right)}^{\alpha }-p-q\right)d\mu (\mathit{x})}}{\alpha (\alpha -1)},}\hfill \\ {\displaystyle \int \left(plog\left(\frac{2p}{p+q}\right)+qlog\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x}),\alpha =0}\hfill \\ {\displaystyle \int (p+q)log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\alpha =1}\hfill \end{array}\right.$	$\left\{\begin{array}{c}\frac{1}{\gamma -1}log{\left[{\displaystyle \int \overline{p}{\left({\displaystyle \frac{p+q}{2p}}\right)}^{\gamma }d\mu (\mathit{x})\int \overline{q}{\left({\displaystyle \frac{p+q}{2q}}\right)}^{\gamma }d\mu (\mathit{x})}\right]}^{\frac{1}{\gamma }}\hfill \\ {\displaystyle \int \left(\overline{p}log\left(\frac{2p}{p+q}\right)+\overline{q}log\left(\frac{2q}{p+q}\right)\right)d\mu (\mathit{x}),\gamma =0}\hfill \\ {\displaystyle \int (\overline{p}+\overline{q})log\left(\frac{p+q}{2\sqrt{pq}}\right)d\mu (\mathit{x}),\gamma =1.}\hfill \end{array}\right.$

Table 4. Basic Alpha- and Beta-divergences and directly generated corresponding Gamma-divergences (see also Table 3 how the Gamma-divergences can be expressed by power means).

Divergence ${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})}$ or ${\displaystyle D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})}$	Gamma-divergence ${\displaystyle D_{AG}^{(\gamma )}(c\mathbf{P}||c\mathbf{Q})}$ and ${\displaystyle D_{BG}^{(\gamma )}(c\mathbf{P}||c\mathbf{Q})}$
${\displaystyle \frac{1}{\alpha (1-\alpha )}\int \left(\alpha p+(1-\alpha )q-p^{\alpha }{q}^{1-\alpha }\right)d\mu (\mathit{x})}$	${\displaystyle -log\frac{{\displaystyle {\left(\int p^{\gamma }{q}^{1-\gamma }d\mu (\mathit{x})\right)}^{1/(\gamma (1-\gamma ))}}}{{\displaystyle {\left(\int pd\mu (\mathit{x})\right)}^{1/(1-\gamma )}{\left(\int qd\mu (\mathit{x})\right)}^{1/\gamma }}}}$
${\displaystyle \frac{1}{\beta (\beta -1)}\int \left(p^{\beta }+(\beta -1)q^{\beta }-\beta p{q}^{\beta -1}\right)d\mu (\mathit{x})}$	${\displaystyle -log\frac{{\displaystyle {\left(\int p{q}^{\gamma -1}d\mu (\mathit{x})\right)}^{1/(\gamma -1)}}}{{\displaystyle {\left(\int p^{\gamma }d\mu (\mathit{x})\right)}^{1/(\gamma (\gamma -1))}{\left(\int q^{\gamma }d\mu (\mathit{x})\right)}^{1/\gamma }}}}$
${\displaystyle \frac{1}{\alpha (1-\alpha )}\int \left(p+q-p^{\alpha }{q}^{1-\alpha }-p^{1-\alpha }{q}^{\alpha }\right)d\mu (\mathit{x})}$	${\displaystyle \frac{-1}{\gamma (1-\gamma )}log\frac{{\displaystyle \left(\int p^{\gamma }{q}^{1-\gamma }d\mu (\mathit{x})\right)\left(\int p^{1-\gamma }{q}^{\gamma }d\mu (\mathit{x})\right)}}{{\displaystyle \int pd\mu (\mathit{x})\int qd\mu (\mathit{x})}}}$
${\displaystyle \frac{1}{\beta -1}\int \left(p^{\beta }+q^{\beta }-p{q}^{\beta -1}-p^{\beta -1}q\right)d\mu (\mathit{x})}$	${\displaystyle \frac{-1}{\gamma -1}log\frac{{\displaystyle \left(\int p{q}^{\gamma -1}d\mu (\mathit{x})\right)\left(\int p^{\gamma -1}qd\mu (\mathit{x})\right)}}{{\displaystyle \left(\int p^{\gamma }d\mu (\mathit{x})\right)\left(\int q^{\gamma }d\mu (\mathit{x})\right)}}}$
${\displaystyle \frac{{\displaystyle \int \left({\displaystyle \frac{\alpha }{2}p+(1-\frac{\alpha }{2})q-{\left({\displaystyle \frac{p+q}{2}}\right)}^{\alpha }{q}^{\alpha -1}}\right)d\mu (\mathit{x})}}{\alpha (1-\alpha )}}$	${\displaystyle \frac{-1}{\gamma (1-\gamma )}log\frac{{\displaystyle \int {\left({\displaystyle \frac{p+q}{2}}\right)}^{\gamma -1}{q}^{\gamma -1}d\mu (\mathit{x})}}{{\displaystyle {\left(\int pd\mu (\mathit{x})\right)}^{\gamma /2}{\left(\int qd\mu (\mathit{x})\right)}^{1-\gamma /2}}}}$
${\displaystyle \frac{1}{1-\alpha }\int (p-q){\left({\displaystyle 1-\frac{p+q}{2q}}\right)}^{\alpha -1}d\mu (\mathit{x})}$	${\displaystyle \frac{1}{1-\gamma }log\frac{{\displaystyle \int pd\mu (\mathit{x})\int q{\left({\displaystyle \frac{p+q}{2q}}\right)}^{\gamma -1}d\mu (\mathit{x})}}{{\displaystyle \int qd\mu (\mathit{x})\int p{\left(\frac{p+q}{2q}\right)}^{\gamma -1}d\mu (\mathit{x})}}}$
${\displaystyle \frac{{\displaystyle \int \left(p+q-(p^{1-\alpha }+q^{1-\alpha }){\left({\displaystyle \frac{p+q}{2}}\right)}^{\alpha }\right)d\mu (\mathit{x})}}{\alpha (1-\alpha )}}$	${\displaystyle log{\left(\frac{{\displaystyle \int p^{1-\gamma }{\left(\frac{p+q}{2}\right)}^{\gamma }d\mu (\mathit{x})\int q^{1-\gamma }{\left(\frac{p+q}{2}\right)}^{\gamma }d\mu (\mathit{x})}}{{\displaystyle \int pd\mu (\mathit{x})\int qd\mu (\mathit{x})}}\right)}^{\frac{1}{\gamma (\gamma -1)}}}$

Table 4. Basic Alpha- and Beta-divergences and directly generated corresponding Gamma-divergences (see also Table 3 how the Gamma-divergences can be expressed by power means).

Divergence ${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})}$ or ${\displaystyle D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})}$	Gamma-divergence ${\displaystyle D_{AG}^{(\gamma )}(c\mathbf{P}||c\mathbf{Q})}$ and ${\displaystyle D_{BG}^{(\gamma )}(c\mathbf{P}||c\mathbf{Q})}$
${\displaystyle \frac{1}{\alpha (1-\alpha )}\int \left(\alpha p+(1-\alpha )q-p^{\alpha }{q}^{1-\alpha }\right)d\mu (\mathit{x})}$	${\displaystyle -log\frac{{\displaystyle {\left(\int p^{\gamma }{q}^{1-\gamma }d\mu (\mathit{x})\right)}^{1/(\gamma (1-\gamma ))}}}{{\displaystyle {\left(\int pd\mu (\mathit{x})\right)}^{1/(1-\gamma )}{\left(\int qd\mu (\mathit{x})\right)}^{1/\gamma }}}}$
${\displaystyle \frac{1}{\beta (\beta -1)}\int \left(p^{\beta }+(\beta -1)q^{\beta }-\beta p{q}^{\beta -1}\right)d\mu (\mathit{x})}$	${\displaystyle -log\frac{{\displaystyle {\left(\int p{q}^{\gamma -1}d\mu (\mathit{x})\right)}^{1/(\gamma -1)}}}{{\displaystyle {\left(\int p^{\gamma }d\mu (\mathit{x})\right)}^{1/(\gamma (\gamma -1))}{\left(\int q^{\gamma }d\mu (\mathit{x})\right)}^{1/\gamma }}}}$
${\displaystyle \frac{1}{\alpha (1-\alpha )}\int \left(p+q-p^{\alpha }{q}^{1-\alpha }-p^{1-\alpha }{q}^{\alpha }\right)d\mu (\mathit{x})}$	${\displaystyle \frac{-1}{\gamma (1-\gamma )}log\frac{{\displaystyle \left(\int p^{\gamma }{q}^{1-\gamma }d\mu (\mathit{x})\right)\left(\int p^{1-\gamma }{q}^{\gamma }d\mu (\mathit{x})\right)}}{{\displaystyle \int pd\mu (\mathit{x})\int qd\mu (\mathit{x})}}}$
${\displaystyle \frac{1}{\beta -1}\int \left(p^{\beta }+q^{\beta }-p{q}^{\beta -1}-p^{\beta -1}q\right)d\mu (\mathit{x})}$	${\displaystyle \frac{-1}{\gamma -1}log\frac{{\displaystyle \left(\int p{q}^{\gamma -1}d\mu (\mathit{x})\right)\left(\int p^{\gamma -1}qd\mu (\mathit{x})\right)}}{{\displaystyle \left(\int p^{\gamma }d\mu (\mathit{x})\right)\left(\int q^{\gamma }d\mu (\mathit{x})\right)}}}$
${\displaystyle \frac{{\displaystyle \int \left({\displaystyle \frac{\alpha }{2}p+(1-\frac{\alpha }{2})q-{\left({\displaystyle \frac{p+q}{2}}\right)}^{\alpha }{q}^{\alpha -1}}\right)d\mu (\mathit{x})}}{\alpha (1-\alpha )}}$	${\displaystyle \frac{-1}{\gamma (1-\gamma )}log\frac{{\displaystyle \int {\left({\displaystyle \frac{p+q}{2}}\right)}^{\gamma -1}{q}^{\gamma -1}d\mu (\mathit{x})}}{{\displaystyle {\left(\int pd\mu (\mathit{x})\right)}^{\gamma /2}{\left(\int qd\mu (\mathit{x})\right)}^{1-\gamma /2}}}}$
${\displaystyle \frac{1}{1-\alpha }\int (p-q){\left({\displaystyle 1-\frac{p+q}{2q}}\right)}^{\alpha -1}d\mu (\mathit{x})}$	${\displaystyle \frac{1}{1-\gamma }log\frac{{\displaystyle \int pd\mu (\mathit{x})\int q{\left({\displaystyle \frac{p+q}{2q}}\right)}^{\gamma -1}d\mu (\mathit{x})}}{{\displaystyle \int qd\mu (\mathit{x})\int p{\left(\frac{p+q}{2q}\right)}^{\gamma -1}d\mu (\mathit{x})}}}$
${\displaystyle \frac{{\displaystyle \int \left(p+q-(p^{1-\alpha }+q^{1-\alpha }){\left({\displaystyle \frac{p+q}{2}}\right)}^{\alpha }\right)d\mu (\mathit{x})}}{\alpha (1-\alpha )}}$	${\displaystyle log{\left(\frac{{\displaystyle \int p^{1-\gamma }{\left(\frac{p+q}{2}\right)}^{\gamma }d\mu (\mathit{x})\int q^{1-\gamma }{\left(\frac{p+q}{2}\right)}^{\gamma }d\mu (\mathit{x})}}{{\displaystyle \int pd\mu (\mathit{x})\int qd\mu (\mathit{x})}}\right)}^{\frac{1}{\gamma (\gamma -1)}}}$

The asymmetric Beta-Gamma-divergence has the following properties [30,35]:

• $D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})\ge 0$. The equality holds if and only if $\mathbf{P}=c\mathbf{Q}$ for a positive constant c.
• It is scale invariant, that is, $D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{BG}^{(\gamma )}(c_1\mathbf{P}||c_2\mathbf{Q})$, for arbitrary positive scaling constants $c_1,c_2$.
• As $\gamma \to 1$, the Gamma-divergence becomes the Kullback–Leibler divergence:

  $${\displaystyle \underset{\gamma \to 1}{lim}{D}_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{KL}(\overline{\mathbf{P}}||\overline{\mathbf{Q}})=\int \overline{p}log\frac{\overline{p}}{\overline{q}}d\mu (\mathit{x})}$$

  (82)

  where $\overline{p}=p/\int pd\mu (\mathit{x})$ and $\overline{q}=q/\int qd\mu (\mathit{x})$.
• For $\gamma \to 0$, the Gamma-divergence can be expressed as follows

  $${\displaystyle \underset{\gamma \to 0}{lim}{D}_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\int log\frac{q(\mathit{x})}{p(\mathit{x})}d\mu (\mathit{x})+log\left(\int \frac{p(\mathit{x})}{q(\mathit{x})}d\mu (\mathit{x})\right)}$$

  (83)

  For the discrete Gamma divergence we have the corresponding formula

  $${\displaystyle D_{BG}^{(0)}(\mathbf{P}||\mathbf{Q})=\frac{1}{n}\sum _{i=1}^n\left(log\frac{q_i}{p_i}\right)+log\left(\sum _{i=1}^n\frac{p_i}{q_i}\right)-log(n)=log\frac{{\displaystyle \frac{1}{n}\sum _{i=1}^n\frac{p_i}{q_i}}}{{\left({\displaystyle \prod _{i=1}^n\frac{p_i}{q_i}}\right)}^{1/n}}}$$

  (84)

Similarly to the Alpha and Beta-divergences, we can also define the symmetric Beta-Gamma-divergence as

$$D_{BGS}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})+D_{BG}^{(\gamma )}(\mathbf{Q}||\mathbf{P})=\frac{1}{1-\gamma }log\left[\frac{\left({\displaystyle \int p^{\gamma -1}qd\mu (\mathit{x})}\right)\left({\displaystyle \int p{q}^{\gamma -1}d\mu (\mathit{x})}\right)}{\left({\displaystyle \int p^{\gamma }d\mu (\mathit{x})}\right)\left({\displaystyle \int q^{\gamma }d\mu (\mathit{x})}\right)}\right]$$

The symmetric Gamma-divergence has similar properties to the asymmetric Gamma-divergence:

• $D_{BGS}^{(\gamma )}(\mathbf{P}||\mathbf{Q})\ge 0$. The equality holds if and only if $\mathbf{P}=c\mathbf{Q}$ for a positive constant c, in particular, $p=q,\forall i$.
• It is scale invariant, that is,

  $$D_{BGS}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{BGS}^{(\gamma )}(c_1\mathbf{P}||c_2\mathbf{Q})$$

  (85)

  for arbitrary positive scaling constants $c_1,c_2$.
• For $\gamma \to 1$, it is reduced to a special form of the symmetric Kullback–Leibler divergence (also called the J-divergence)

  $${\displaystyle \underset{\gamma \to 1}{lim}{D}_{BGS}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\int (\overline{p}-\overline{q})log\frac{\overline{p}}{\overline{q}}d\mu (\mathit{x})}$$

  (86)

  where $\overline{p}=p/\int pd\mu (\mathit{x})$ and $\overline{q}=q/\int qd\mu (\mathit{x})$.
• For $\gamma =0$, we obtain a simple divergence expressed by weighted arithmetic means

  $${\displaystyle D_{BGS}^{(0)}(\mathbf{P}||\mathbf{Q})=log\left({\displaystyle \int w\frac{p(\mathit{x})}{q(\mathit{x})}d\mu (\mathit{x})\int w\frac{q(\mathit{x})}{p(\mathit{x})}d\mu (\mathit{x})}\right)}$$

  (87)

  where weight function $w>0$ is such that $\int wd\mu (\mathit{x})=1$.
  For the discrete Beta-Gamma divergence (or simply the Gamma divergence), we obtain divergence

  $$\begin{array}{ccc}\hfill D_{BGS}^{(0)}(\mathbf{P}||\mathbf{Q})& =\hfill & \hfill {\displaystyle log\left(\sum _{i=1}^n\frac{p_i}{q_i}\sum _{i=1}^n\frac{q_i}{p_i}\right)-log{(n)}^2}\\ & =\hfill & \hfill log\left(\left(\frac{1}{n}\sum _{i=1}^n\frac{p_i}{q_i}\right)\left(\frac{1}{n}\sum _{i=1}^n\frac{q_i}{p_i}\right)\right)\\ & =\hfill & \hfill log\left({\displaystyle M_1\left\{\frac{p_i}{q_i}\right\}{M}_1\left\{\frac{q_i}{p_i}\right\}}\right)\end{array}$$

  It is interesting to note that for $n\to \infty$ the discrete symmetric Gamma-divergence can be expressed by expectation functions $D_{BGS}^{(1)}(\mathbf{P}||\mathbf{Q})=log(E\{\mathit{u}\}E\{{\mathit{u}}^{-1}\})$, where $u_i=\{p_i/q_i\}$ and $u_i^{-1}=\{q_i/p_i\}$.
• For $\gamma =2$, the asymmetric Gamma-divergences (equal to a symmetric Gamma-divergence) is reduced to Cauchy–Schwarz divergence, introduced by Principe [83]

  $$\begin{array}{cccc}\hfill D_{BG}^{(2)}(\mathbf{P}||\mathbf{Q})& & \hfill =& D_{BG}^{(2)}(\mathbf{Q}||\mathbf{P})=\frac{1}{2}{D}_{BGS}^{(2)}(\mathbf{P}||\mathbf{Q})=\frac{1}{2}{D}_{CS}(\mathbf{P}||\mathbf{Q})\hfill \\ & & \hfill =& {\displaystyle -log\frac{{\displaystyle \int p(\mathit{x})q(\mathit{x})d\mu (\mathit{x})}}{{\left({\displaystyle \int p^2(\mathit{x})d\mu (\mathit{x})}\right)}^{1/2}{\left({\displaystyle \int q^2(\mathit{x})d\mu (\mathit{x})}\right)}^{1/2}}}\hfill \end{array}$$

It should be noted that the basic asymmetric Beta-Gamma divergence (derived from the Beta-divergence) is exactly equivalent to the Gamma divergence defined in [35], while Alpha-Gamma divergences (derived from the family of Alpha-divergences) have different expressions but they are similar in terms of properties.

5. Relationships for Asymmetric Divergences and their Unified Representation

The fundamental relationships and analogies for the generalized divergences discussed in this paper are summarized in Table 5.

The basic difference among Alpha- Beta and Gamma divergences is their scaling properties

$$D_A^{(\alpha )}(c\mathbf{P}||c\mathbf{Q})=c{D}_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})$$

(88)

$$D_B^{(\beta )}(c\mathbf{P}||c\mathbf{Q})=c^{\beta }{D}_B^{(\beta )}(\mathbf{P}||\mathbf{Q})$$

(89)

Table 5. Fundamental asymmetric divergences and their relationships. We can obtain the Beta-divergence from the Alpha-divergence via transformations $p\to p^{\beta },q\to q^{\beta }\mathrm{and}\alpha ={\beta }^{-1}$ and the Alpha-divergence from the Beta-divergence via dual transformations: $p\to p^{\alpha },q\to q^{\alpha }and\beta ={\alpha }^{-1}$. Furthermore, we can generate the Gamma-divergences from the Alpha- and Beta- divergences via transformations $c_0\int p^{c_1}(\mathit{x})q^{c_2}(\mathit{x})d\mu (\mathit{x})\to c_{0}log\int p^{c_1}(\mathit{x})q^{c_2}(\mathit{x})d\mu (\mathit{x})$. Moreover, we can generate Beta-Gamma divergences from the Alpha-Gamma divergence via transformations: $p\to p^{{\gamma }_{new}},q\to q^{{\gamma }_{new}}\mathrm{and}\gamma ={\gamma }_{new}^{-1}$.

Divergence name	Formula
Alpha	${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\frac{\int (p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q)d\mu (\mathit{x})}{\alpha (\alpha -1)}}$
Beta	${\displaystyle D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=\frac{\int (p^{\beta }+(\beta -1)q^{\beta }-\beta p{q}^{\beta -1})d\mu (\mathit{x})}{\beta (\beta -1)}}$
Gamma	${\displaystyle D_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\frac{log(\int p^{\gamma }{q}^{1-\gamma }d\mu (\mathit{x}))-\gamma log(\int pd\mu (\mathit{x}))+(\gamma -1)log(\int qd\mu (\mathit{x}))}{\gamma (\gamma -1)}}$
	${\displaystyle =\frac{1}{\gamma -1}log{\left(\int \overline{q}{\left(\frac{\overline{p}}{\overline{q}}\right)}^{\gamma }d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }}}$
	${\displaystyle D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\frac{log(\int p^{\gamma }d\mu (\mathit{x}))+(\gamma -1)log(\int q^{\gamma }d\mu (\mathit{x}))-\gamma log(\int p{q}^{\gamma -1}d\mu (\mathit{x}))}{\gamma (\gamma -1)}}$
	${\displaystyle =\frac{1}{1-\gamma }log\left(\int {\tilde{q}}^{\gamma }{\left(\frac{\tilde{p}}{\tilde{q}}\right)}^{}d\mu (\mathit{x})\right)}$ ))
Alpha-Rényi	${\displaystyle D_{AR}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\frac{log(\int (p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q)d\mu (\mathit{x})+1)}{\alpha (\alpha -1)}}$
Bregman	${\displaystyle D_{\Phi }(\mathbf{P}||\mathbf{Q})=\int \left(\Phi (p)-\Phi (q)-\frac{\delta \Phi }{\delta q}(p-q)\right)d\mu (\mathit{x})}$, (see Eqation(57))
Csiszár-Morimoto	${\displaystyle D_f(\mathbf{P}||\mathbf{Q})=\int qf\left(\frac{p}{q}\right)d\mu (\mathit{x})}$, (see Equation (57)) for $f(u)=\Phi {(t)}_{|t=u}$)

Table 5. Fundamental asymmetric divergences and their relationships. We can obtain the Beta-divergence from the Alpha-divergence via transformations $p\to p^{\beta },q\to q^{\beta }\mathrm{and}\alpha ={\beta }^{-1}$ and the Alpha-divergence from the Beta-divergence via dual transformations: $p\to p^{\alpha },q\to q^{\alpha }and\beta ={\alpha }^{-1}$. Furthermore, we can generate the Gamma-divergences from the Alpha- and Beta- divergences via transformations $c_0\int p^{c_1}(\mathit{x})q^{c_2}(\mathit{x})d\mu (\mathit{x})\to c_{0}log\int p^{c_1}(\mathit{x})q^{c_2}(\mathit{x})d\mu (\mathit{x})$. Moreover, we can generate Beta-Gamma divergences from the Alpha-Gamma divergence via transformations: $p\to p^{{\gamma }_{new}},q\to q^{{\gamma }_{new}}\mathrm{and}\gamma ={\gamma }_{new}^{-1}$.

Divergence name	Formula
Alpha	${\displaystyle D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\frac{\int (p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q)d\mu (\mathit{x})}{\alpha (\alpha -1)}}$
Beta	${\displaystyle D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=\frac{\int (p^{\beta }+(\beta -1)q^{\beta }-\beta p{q}^{\beta -1})d\mu (\mathit{x})}{\beta (\beta -1)}}$
Gamma	${\displaystyle D_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\frac{log(\int p^{\gamma }{q}^{1-\gamma }d\mu (\mathit{x}))-\gamma log(\int pd\mu (\mathit{x}))+(\gamma -1)log(\int qd\mu (\mathit{x}))}{\gamma (\gamma -1)}}$
	${\displaystyle =\frac{1}{\gamma -1}log{\left(\int \overline{q}{\left(\frac{\overline{p}}{\overline{q}}\right)}^{\gamma }d\mu (\mathit{x})\right)}^{\frac{1}{\gamma }}}$
	${\displaystyle D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=\frac{log(\int p^{\gamma }d\mu (\mathit{x}))+(\gamma -1)log(\int q^{\gamma }d\mu (\mathit{x}))-\gamma log(\int p{q}^{\gamma -1}d\mu (\mathit{x}))}{\gamma (\gamma -1)}}$
	${\displaystyle =\frac{1}{1-\gamma }log\left(\int {\tilde{q}}^{\gamma }{\left(\frac{\tilde{p}}{\tilde{q}}\right)}^{}d\mu (\mathit{x})\right)}$ ))
Alpha-Rényi	${\displaystyle D_{AR}^{(\alpha )}(\mathbf{P}||\mathbf{Q})=\frac{log(\int (p^{\alpha }{q}^{1-\alpha }-\alpha p+(\alpha -1)q)d\mu (\mathit{x})+1)}{\alpha (\alpha -1)}}$
Bregman	${\displaystyle D_{\Phi }(\mathbf{P}||\mathbf{Q})=\int \left(\Phi (p)-\Phi (q)-\frac{\delta \Phi }{\delta q}(p-q)\right)d\mu (\mathit{x})}$, (see Eqation(57))
Csiszár-Morimoto	${\displaystyle D_f(\mathbf{P}||\mathbf{Q})=\int qf\left(\frac{p}{q}\right)d\mu (\mathit{x})}$, (see Equation (57)) for $f(u)=\Phi {(t)}_{|t=u}$)

$$D_G^{(\gamma )}(c\mathbf{P}||c\mathbf{Q})=D_G^{(\gamma )}(\mathbf{P}||\mathbf{Q})$$

(90)

for any $c>0$.

Note that the values of all Alpha-divergences are proportional to a scaling factor and this scaling is independent of the parameter α, while for all Beta-divergences this scaling factor is dependent on the value of the parameter β exponentially. Only in a special case for $\beta =0$ (Itakura–Saito distance), the Beta-divergences are invariant to a scaling factor, while the Gamma-divergences is invariant to a scaling factor for any value of the parameter γ.

On the basis of our analysis and discussion in the previous sections, we come to conclusion that the wide class of asymmetric divergences can be described by the following generalized function:

$$D_{AC}^{(\alpha ,\beta ,r)}(\mathbf{P}||\mathbf{Q})=\frac{1}{\alpha (\beta -1)\lambda r}\left[{\left({\tilde{D}}_{AC}^{(\alpha ,\beta )}+1\right)}^r-1\right]$$

(91)

where $r>0$ and

$$\begin{array}{ccc}\hfill {\tilde{D}}_{AC}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})& =& \int \left(\alpha p^{\lambda }+(\beta -1)q^{\lambda }-\lambda p^{\alpha }{q}^{\beta -1}\right)d\mu (\mathit{x})\hfill \end{array}$$

(92)

with $\alpha \ne 0,\beta \ne 1$ and $\lambda =\alpha +\beta -1\ne 0$.

Note that such defined function has similar structure to generalized logarithm (3). Moreover, it should be noted that the above function is a divergence only for such set of parameters for which ${\tilde{D}}_{AC}^{(\alpha ,\beta )}(\mathbf{P}\parallel \mathbf{Q})\ge 0$ and equals zero if and only if $\mathbf{P}=\mathbf{Q}$.

In the special cases, we obtain the following divergences:

• For $r=\lambda =1$ and $\alpha +\beta =2$, we obtain the Alpha-divergence (4)
• For $r=1$ and $\alpha =1$ $(\beta =\lambda )$, we obtain the Beta-divergence (46)
• For $r\to 0$ and $\alpha +\beta =2$ $(\lambda =1)$, we obtain the Alpha-Rényi divergence (25), which for normalized probabilities densities: $\overline{p}=p/(\int pd\mu (\mathit{x}))$ and $\overline{q}=q/(\int qd\mu (\mathit{x}))$ reduces to the Alpha-Gamma divergence.
• And finally, for $r\to 0$ and $\alpha =1$ $(\beta =\lambda =\gamma$) and the normalized densities: $\tilde{p}=p/{(\int p^{\gamma }d\mu (\mathit{x}))}^{1/\gamma }$ and $\tilde{q}=q/{(\int q^{\gamma }d\mu (\mathit{x}))}^{1/\gamma }$, we obtain the Gamma divergence (referred in this paper as the Beta-Gamma divergence) (72), proposed recently by Fujisawa and Eguchi [35].

The function (91) has many interesting general properties. However, this is out of the scope of this paper.

Duality

Note that we can easily establish the following dualities for the Alpha and Alpha-Gamma divergences:

$$D_A^{(\alpha )}(\mathbf{P}||\mathbf{Q})=c^{-1}{D}_A^{(1-\alpha )}(c\mathbf{Q}||c\mathbf{P}),D_{AG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{AG}^{(1-\gamma )}(c_1\mathbf{Q}||c_2\mathbf{P})$$

(93)

However, in order to establish duality for the Beta-divergence, we need represent it in slightly more general form as two set parameters divergence:

$$D_{AB}^{({\beta }_1,{\beta }_2)}(\mathbf{P}||\mathbf{Q})=\frac{1}{{\beta }_1{\beta }_2({\beta }_1+{\beta }_2)}\int \left({\beta }_1{p}^{{\beta }_1+{\beta }_2}+{\beta }_2{q}^{{\beta }_1+{\beta }_2}-({\beta }_1+{\beta }_2)p^{{\beta }_1}{q}^{{\beta }_2}\right)d\mu (\mathit{x})$$

(94)

In this case, we can express a duality of the Beta-divergence as

$$D_B^{(\beta )}(\mathbf{P}||\mathbf{Q})=D_{AB}^{(1,\beta -1)}(\mathbf{P}||\mathbf{Q})=c^{-\beta }{D}_{AB}^{(\beta -1,1)}(c\mathbf{Q}||c\mathbf{P})$$

(95)

or more generally

$$D_{AB}^{({\beta }_1,{\beta }_2)}(\mathbf{P}||\mathbf{Q})=c^{-({\beta }_1+{\beta }_2)}{D}_{AB}^{({\beta }_2,{\beta }_1)}(c\mathbf{Q}||c\mathbf{P})$$

(96)

Analogously, we define family of the asymmetric Gamma divergences in more general form as a two-set parameter divergence:

$$\begin{array}{c}\hfill D_{ABG}^{({\gamma }_1,{\gamma }_2)}(\mathbf{P}\parallel \mathbf{Q})=log\frac{{\left({\displaystyle \int p^{{\gamma }_1+{\gamma }_2}d\mu (\mathit{x})}\right)}^{\frac{1}{{\gamma }_2({\gamma }_1+{\gamma }_2)}}{\left({\displaystyle \int q^{{\gamma }_1+{\gamma }_2}d\mu (\mathit{x})}\right)}^{\frac{1}{{\gamma }_1({\gamma }_1+{\gamma }_2)}}}{{\left({\displaystyle \int p^{{\gamma }_1}{q}^{{\gamma }_2}d\mu (\mathit{x})}\right)}^{\frac{1}{{\gamma }_1{\gamma }_2}}}\end{array}$$

(97)

Hence, we express duality property for the Beta-Gamma (basic Gamma) -divergence as follows

$$D_{BG}^{(\gamma )}(\mathbf{P}||\mathbf{Q})=D_{ABG}^{(1,\gamma -1)}(\mathbf{P}||\mathbf{Q})=D_{ABG}^{(\gamma -1,1)}(c_1\mathbf{Q}||c_2\mathbf{P})$$

(98)

or more generally

$$D_{ABG}^{({\gamma }_1,{\gamma }_2)}(\mathbf{P}||\mathbf{Q})=D_{ABG}^{({\gamma }_2,{\gamma }_1)}(c_1\mathbf{Q}||c_2\mathbf{P})$$

(99)

5.1. Conclusions and Discussion

The main goal of this paper was to establish and widen bridges between several classes of divergences. The main results are summarized in Table 2, Table 3, Table 4 and Table 5. We have extended and unified several important classes of divergence and provided links and correspondences among them. The parametric families of the generalized divergences presented in this paper, especially the Alpha-, Alpha-Rényi and Beta-divergences, enable us to consider smooth connections of various pairs of well-known and frequently used fundamental divergences under one “umbrella”. For example, the family of Beta-divergences smoothly connects the squared Euclidean distance ($L_2$-norm) with a generalized extended version of Itakura–Saito like distances and through the the generalized KL I-divergences. In fact, we have shown how to generate the whole family of the extended robust Itakura–Saito like distances from Beta-divergences for $\beta =0$, which are scale invariant and can be used in many potential applications, especially in speech processing.

Furthermore, we have shown that two different families of Gamma-divergences can be generated one from Beta-divergences and the other one from Alpha-divergences. The Gamma-divergences characterized by the global scale invariance (independent of value of parameter γ) and high robustness. We have reviewed and extended fundamental properties of the Gamma-divergences. Moreover, our approach allows us to generate a wide class of new divergences, especially Gamma-divergences and family of extended Itakura–Saito like distances. Special emphasis is given to divergences for singular values of tuning parameters $0,1$.

These families have many desirable properties such as flexibility, robustness to outliers and they involve only one single tuning parameter (α, β, or γ). An insight using information geometry may further elucidates the fundamental structures of such divergences and geometry [1,3,7].

In comparison to previous closely related works, we explicitly investigated three wide class of divergences (Alpha, Beta- Gama), discussed their properties and links to Csiszár–Morimoto and Bregman divergences. Some provided properties, especially for Gamma-divergences are new.

The properties of divergences discussed in this paper allowed us to develop a wide family of robust algorithms for Independent Component Analysis (ICA) and Nonnegative Matrix Factorization (NMF) presented in separated recent works [30,31,32,33,34,68,69,70,71]. They have great potential in many applications, especially in blind separation of statistically dependent sparse and/or smooth sources and in feature extractions and classification problems [30].

One of the important and still open problem (not discussed in this paper) is how to set tuning parameters $\alpha ,\beta$ and γ depending on distributions of available data sets and noise or outliers. Some recent works address this problem [69,71,84]. Usually, there is some trade off between robustness and efficiency [35,37,71]. For example, if the tuning parameter γ is large, then the robustness of the Gamma divergence will be strong but the efficiency could be lower, and vice versa, if the tuning parameter has small positive value, then the robustness of the method would be not strong but the efficiency will be higher [35]. The relation between the efficiency and the tuning parameters was discussed by several authors (see Jones et al. [85], Basu et al. [67], and Fujisawa and Eguchi [35]).