Mirror Descent and Exponentiated Gradient Algorithms Using Trace-Form Entropies

Abstract

This paper introduces a broad class of Mirror Descent (MD) and Generalized Exponentiated Gradient (GEG) algorithms derived from trace-form entropies defined via deformed logarithms. Leveraging these generalized entropies yields MD and GEG algorithms with improved convergence behavior, robustness against vanishing and exploding gradients, and inherent adaptability to non-Euclidean geometries through mirror maps. We establish deep connections between these methods and Amari’s natural gradient, revealing a unified geometric foundation for additive, multiplicative, and natural gradient updates. Focusing on the Tsallis, Kaniadakis, Sharma–Taneja–Mittal, and Kaniadakis–Lissia–Scarfone entropy families, we show that each entropy induces a distinct Riemannian metric on the parameter space, leading to GEG algorithms that preserve the natural statistical geometry. The tunable parameters of deformed logarithms enable adaptive geometric selection, providing enhanced robustness and convergence over classical Euclidean optimization. Overall, our framework unifies key first-order MD optimization methods under a single information-geometric perspective based on generalized Bregman divergences, where the choice of entropy determines the underlying metric and dual geometric structure.

This paper is dedicated to Professor Shun-Ichi AMARI in honor of his 90th birthday.

1. Introduction

Mirror descent (MD), initially proposed by Nemirovsky and Yudin [1], has become an increasingly popular topic in optimization, artificial intelligence, and machine learning domains [2,3,4,5,6]. Its profound success stems not merely from its algorithmic efficiency, but also from deep mathematical connections to information geometry and the natural statistical structure underlying optimization problems. These connections, particularly to Amari’s Natural Gradient (NG) method, reveal that effective optimization is fundamentally about respecting the intrinsic geometry of the parameter space rather than imposing artificial Euclidean constraints [7,8,9].

The central motivation for our research emerges from a fundamental insight in information geometry: optimized learning algorithms should adapt to the Fisher information metric of the underlying statistical manifold. This principle, pioneered by Amari in the context of neural networks, establishes that the steepest descent direction on a statistical manifold is not the Euclidean gradient, but rather the natural gradient, the direction that accounts for the curvature induced by the Fisher information matrix [10,11].

1.1. The Information Geometry Perspective

The connection between Mirror Descent and Natural Gradient runs deeper than algorithmic similarity—it represents a fundamental mathematical equivalence that has been rigorously established [2,12,13]. Our work extends this principle by showing that trace-form entropies induce natural Fisher—like metrics that can be even more appropriate for specific problem structures. Through deformed logarithms, we can systematically explore the space of possible geometries and the possibility of discovering optimal choices of hyperparameters for given data distributions [14,15].

1.2. Challenge of Geometric Selection

While the power of geometric optimization is well established, a fundamental challenge remains: how to select the appropriate geometry for a given optimization problem? Classical approaches require domain expertise and manual tuning, limiting their applicability. Our approach addresses this through parameterized entropy families, where hyperparameters control the geometric structure [2].

The theoretical foundation of our approach rests on the connection between Bregman divergences and exponential families established in information geometry [8]. This connection indicates that choosing a generalized entropy is equivalent to selecting an appropriate exponential family structure for specific optimization problems. See also [16] for an extension of the logarithmic divergences extending to Bregman divergences. Our deformed logarithms allow us to systematically explore the space of possible exponential families, enabling discovery of optimal statistical models.

The Exponentiated Gradient (EG) and its extensions emerge as a specific and powerful instantiation of the Mirror Descent framework when the mirror map is constructed from generalized entropies and deformed logarithms. This connection is far from superficial—it represents a fundamental mathematical relationship that unifies additive and multiplicative gradient updates within a single theoretical framework [15,17,18,19,20,21].

It is important to note that Mirror Descent updates can be reparameterized as Gradient Descent in appropriately chosen coordinate systems [2,12]. This reveals that the computational complexity of Natural Gradient can be considerably reduced. Our deformed logarithms approach gives some insight by showing that deformed logarithms naturally induce reparameterizations that preserve geometric structure while enabling efficient computation and provide implicit regularization through the choice of entropy or deformed logarithm.

1.3. Research Contributions and Scope

In this work, we systematically investigate the theoretical foundations and practical implications of employing trace-form entropies and deformed logarithms in the Mirror Descent framework.

Our primary contributions include:

• Mathematical Framework: We establish a comprehensive mathematical foundation connecting generalized entropies, deformed logarithms, and Mirror Descent updates, providing explicit formulations for numerous well-established trace entropy families.
• Algorithmic Innovations: We derive novel Generalized Exponentiated Gradient (GEG) algorithms with generalized multiplicative updates that leverage the flexibility of hyperparameter-controlled deformed logarithms, enabling adaptation to problem geometry.
• The significance of this research extends beyond algorithmic development; it opens new avenues for understanding the geometric foundations of optimization and provides practical tools for addressing increasingly complex machine learning challenges. The unifying theoretical framework connects optimization theory, information geometry, statistical physics, and practical machine learning, opens up new research directions, and provides principled approaches to algorithm design that respect the natural geometric structure of optimization problems.

2. Preliminaries: Mirror Descent (MD) and Standard Exponentiated Gradient (EG) Updates

Notations: Vectors are denoted by boldface lowercase letters, e.g., $\mathit{w}\in {\mathbb{R}}^N$, where for any vector w, we denote its i-th entry by ${\mathit{w}}_i$. For any vectors $\mathit{w},v\in {\mathbb{R}}^N$, we define the Hadamard product as $\mathit{w}\odot v={[{\mathit{w}}_1{v}_1,\dots ,{\mathit{w}}_N{v}_N]}^T$ and ${\mathit{w}}^{\alpha }={[{\mathit{w}}_1^{\alpha },\dots ,{\mathit{w}}_N^{\alpha }]}^T$. All operations for vectors like multiplications and additions are performed componentwise. The function of a vector is also applied for any entry of the vectors, e.g., $f\left(\mathit{w}\right)={[f\left({\mathit{w}}_1\right),f\left({\mathit{w}}_2\right),\dots ,f\left({\mathit{w}}_N\right)]}^T$. The N-dimensional real vector space with nonnegative real numbers is denoted by ${\mathbb{R}}_{+}^N$. We let $\mathit{w}\left(t\right)$ denote the weight or parameter vector as a function of time t. The learning process advances in iterative steps, where during step t we start with the weight vector $\mathit{w}\left(t\right)={\mathit{w}}_t$ and update it to a new vector $\mathit{w}(t+1)={\mathit{w}}_{t+1}$. We define ${\left[x\right]}_{+}=\max \{0,x\}$, and the gradient of a differentiable cost function as ${\nabla }_{\mathit{w}}L\left(\mathit{w}\right)=\mathsf{\partial }L\left(\mathit{w}\right)/\mathsf{\partial }\mathit{w}={[\mathsf{\partial }L\left(\mathit{w}\right)/\mathsf{\partial }{\mathit{w}}_1,\dots ,\mathsf{\partial }L\left(\mathit{w}\right)/\mathsf{\partial }{\mathit{w}}_N]}^T$. In contrast to deformed logarithms defined later, the classical natural logarithm will be denoted by $\ln \left(x\right)$.

2.1. Problem Statement

We consider the constrained optimization problem:

$$\begin{array}{c}\hfill {\mathit{w}}_{t+1}=\underset{\mathit{w}\in {\mathbf{R}}_{+}^N}{\arg \min }\left\{L\left(\mathit{w}\right)+{\displaystyle \frac{1}{\eta }}{D}_F\left(\mathit{w}\right||{\mathit{w}}_t)\right\},\end{array}$$

(1)

where $L\left(\mathit{w}\right)$ is a continuously differentiable loss function, $\eta >0$ is the learning rate and $D_F\left(\mathit{w}\right||{\mathit{w}}_t)$ is the Bregman divergence induced by a strictly convex generating function $F\left(\mathit{w}\right)$ (used here as a regularizer) [2,22]. The Bregman divergence provides the geometric foundation for Mirror Descent algorithms:

$$\begin{array}{c}\hfill D_F\left(\mathit{w}\right||{\mathit{w}}_t)=F\left(\mathit{w}\right)-F\left({\mathit{w}}_t\right)-{(\mathit{w}-{\mathit{w}}_t)}^{T}f\left({\mathit{w}}_t\right),\end{array}$$

(2)

where the generating (or potential) function $F\left(\mathit{w}\right)$ is a continuously-differentiable, strictly convex function defined on the convex domain, while $f\left(\mathit{w}\right)={\nabla }_{\mathit{w}}F\left(\mathit{w}\right)$ is the mirror map, called also the link function, which is a strictly monotonically increasing function. For fundamental properties and for some extensions, see, e.g., [23,24,25,26].

The Bregman divergence measures the difference between $F\left(\mathit{w}\right)$ and its first-order Taylor approximation around ${\mathit{w}}_t$, providing a natural measure of geometric proximity that respects the curvature induced by F. This geometric structure is intimately connected to information geometry—different choices of F correspond to different Riemannian metrics on the parameter manifold. The Bregman divergence $D_F\left(\mathit{w}\right||{\mathit{w}}_t)$ arising from generating (potential) function $F\left(\mathit{w}\right)$ can be viewed as a measure of curvature. The Bregman divergence includes many well-known divergences commonly used in practice, namely, the squared Euclidean distance, Kullback–Leibler divergence (relative entropy), Itakura-Saito distance, beta divergence and many more [10,14,27,28,29].

2.2. Mirror Descent Update Rules and Geometric Interpretation

Setting the gradient of the objective in Equation (1) to zero yields the implicit update:

$$\begin{array}{c}\hfill f\left({\mathit{w}}_{t+1}\right)=f\left({\mathit{w}}_t\right)-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_{t+1}\right),\end{array}$$

(3)

or equivalently

$$\begin{array}{c}\hfill {\mathit{w}}_{t+1}=f^{(-1)}\left[f\left({\mathit{w}}_t\right)-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_{t+1}\right)\right],\end{array}$$

(4)

where $f^{(-1)}$ is inverse function of the link function. Note that when F is separable and continuous, the inverse function $F^{-1}$ is defined globally (by the inverse function theorem). In general, the implicit function theorem only guarantees local inversion of multivariate functions but not the existence of global inverse functions. However, when F is a multivariate convex function of Legendre-type so is its convex conjugate $F^{*}$ and their gradients are reciprocal to each others globally: $\nabla F={(\nabla F^{*})}^{-1}$ and $\nabla F^{*}={(\nabla F)}^{-1}$.

Assuming that ${\nabla }_{\mathit{w}}L\left({\mathit{w}}_{t+1}\right)\approx {\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)$, we obtain the explicit Mirror Descent update [2,5]:

$$\begin{array}{c}{\displaystyle {\mathit{w}}_{t+1}=f^{(-1)}\left[f({\mathit{w}}_t)-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right]}\hfill \end{array}$$

(5)

$$\begin{array}{c}=\nabla F^{(-1)}\left(\nabla F\left({\mathit{w}}_t\right)-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right).\hfill \end{array}$$

(6)

In MD, we map our primal point w to the dual space (through the mapping via the link function $f\left(\mathit{w}\right)=\nabla F\left(\mathit{w}\right)$) and take a step in the direction given by the gradient of the function, then we map back to the primal space by using the inverse function of the link function. The advantage of using Mirror Descent (MD) over Gradient Descent is that it takes into account the geometry of the problem through suitable choice of a link function.

Dual Space Interpretation: Mirror Descent operates by

• Mapping to dual space: $\Theta =f\left(\mathit{w}\right)=\nabla F\left(\mathit{w}\right)$,
• Taking gradient step: ${\Theta }_{t+1}={\Theta }_t-\eta \nabla L\left({\mathit{w}}_t\right)$,
• Mapping back to primal: ${\mathit{w}}_{t+1}=f^{-1}\left({\Theta }_{t+1}\right)$.

This three-step process naturally incorporates problem geometry through the choice of link function f (Figure 1).

For example, consider $F\left(\mathit{w}\right)={\sum }_i{\mathit{w}}_i\log {\mathit{w}}_i$, the Shannon negative entropy. The link function is $f\left(\mathit{w}\right)=\nabla F\left(\mathit{w}\right)={[1+\log {\mathit{w}}_i]}_i$ with inverse map $f^{-1}(\Theta )={\left[{\displaystyle \frac{e^{{\Theta }_i}}{{\sum }_j{e}^{{\Theta }_j}}}\right]}_i$. The corresponding mirror update is the exponentiated gradient update:

$${\mathit{w}}_{t+1}={\mathit{w}}_t\odot \exp \left(-\eta \hspace{0.33em}{\nabla }_{\mathit{w}}\hspace{0.33em}L\left({\mathit{w}}_t\right)\right).$$

This is a standard and useful algorithm for optimization on the probability simplex that is recovered as the mirror descent with respect to the Kullback–Leibler (KL) divergence (a Bregman divergence). The underlying geometric structure is the KL Hessian geometry, an example of dually flat space in information geometry.

Note that when the generating function F is separable across its coordinates (i.e., $F\left(\mathit{w}\right)={\sum }_{i}F\left({\mathit{w}}_i\right)$), the Hessian matrix ${\nabla }^{2}F\left(\mathit{w}\right)$ is diagonal.

2.3. Continuous-Time Formulation and Natural Gradient Connection

The continuous-time limit (as $\Delta t\to 0$) yields the mirror flow ODE:

$$\boxed{{\displaystyle \frac{\mathit{d}\hspace{0.33em}\mathit{f}(\mathit{w}(\mathit{t}))}{\mathit{d}\mathit{t}}}=-\mu {\nabla }_{\mathit{w}}\mathit{L}(\mathit{w}(\mathit{t})),}$$

(7)

where $\mu =\eta /\Delta t>0$ is the learning rate for continuous-time learning, and $f\left(\mathit{w}\right)=\nabla F\left(\mathit{w}\right)$ is a suitably chosen link function [2]. Using the chain rule, we can write mirror flow as follows

$$\begin{array}{c}\hfill {\displaystyle \frac{d\hspace{0.33em}f\hspace{0.33em}\left(\mathit{w}\right)}{dt}}={\displaystyle \frac{d\hspace{0.33em}f\left(\mathit{w}\right)}{d\mathit{w}}}\odot {\displaystyle \frac{d\hspace{0.33em}\mathit{w}}{dt}}=\mathrm{diag}\left\{{\displaystyle \frac{d\hspace{0.33em}f\left(\mathit{w}\right)}{d\mathit{w}}}\right\}{\displaystyle \frac{d\hspace{0.33em}\mathit{w}}{dt}}=-\mu {\nabla }_{\mathit{w}}L\left(\mathit{w}\left(t\right)\right).\end{array}$$

(8)

Hence, we can obtain continuous-time MD update in alternative form:

$$\begin{array}{c}\hfill {\displaystyle \frac{d\hspace{0.33em}\mathit{w}}{dt}}=-\mu \mathrm{diag}\left\{{\left({\displaystyle \frac{d\hspace{0.33em}f\left(\mathit{w}\right)}{d\mathit{w}}}\right)}^{-1}\right\}{\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)=-\mu \hspace{0.33em}{\left[{\nabla }^{2}F\left(\mathit{w}\right)\right]}^{-1}\hspace{0.33em}{\nabla }_{\mathit{w}}L\left(\mathit{w}\left(t\right)\right).\end{array}$$

(9)

This reveals that Mirror Descent in continuous-time is equivalent to Natural Gradient descent with the Riemannian metric $H_F\left(\mathit{w}\right)={\left[{\nabla }^{2}F\left(\mathit{w}\right)\right]}^{-1}$. This connection, established rigorously in [2,13], shows that geometric optimization methods are fundamentally unified.

2.4. Discrete Natural Gradient Form

The discrete version becomes

$$\boxed{{\mathit{w}}_{\mathit{t}+1}=[{\mathit{w}}_{\mathit{t}}-\eta \mathrm{diag}\left\{{\left({\displaystyle \frac{\mathit{d}\hspace{0.33em}\mathit{f}\left({\mathit{w}}_{\mathit{t}}\right)}{\mathit{d}{\mathit{w}}_{\mathit{t}}}}\right)}^{-1}\right\}{\nabla }_{\mathit{w}}\mathit{L}\left({\mathit{w}}_{\mathit{t}}\right){]}_{+}}$$

(10)

where ${\displaystyle \mathrm{diag}\left\{{\left(\frac{d\hspace{0.33em}f\left(\mathit{w}\right)}{d\mathit{w}}\right)}^{-1}\right\}=\mathrm{diag}\left\{{\left(\frac{d\hspace{0.33em}f\left(\mathit{w}\right)}{d{w}_1}\right)}^{-1},\dots ,{\left(\frac{d\hspace{0.33em}f\left(\mathit{w}\right)}{d{w}_N}\right)}^{-1}\right\}}$, which we term Mirror-less Mirror Descent (MMD), representing a first-order approximation to second-order Natural Gradient methods (10) [30].

It should be noted that the above-defined diagonal matrix can be considered as the inverse of the Hessian matrix, if it exists and has positive diagonal entries for a specific set of parameters. The MMD is a special form of Natural Gradient Descent (NGD) [2,7,13].

F. Nielsen provided a geometric interpretation of NG and its connections with the Riemannian gradient, the mirror descent, and the ordinary additive gradient descent [31].

2.5. Canonical Examples and Some Geometric Insights

Case 1: For $F\left(\mathit{w}\right)={\parallel \mathit{w}\parallel }_2^2/2={\displaystyle \frac{1}{2}}{\sum }_{i=1}^N{\mathit{w}}_i^2$ and link function $f\left(\mathit{w}\right)={\nabla }_{\mathit{w}}F\left(\mathit{w}\right)=\mathit{w}$, we obtain the standard (additive) gradient descent

$$\begin{array}{c}\hfill {\displaystyle \frac{d\mathit{w}\left(t\right)}{dt}}=-{\mu }_t{\nabla }_{\mathit{w}}L\left(\mathit{w}\left(t\right)\right)\end{array}$$

(11)

and its time-discrete approximate version

$$\begin{array}{c}\hfill {\mathit{w}}_{t+1}={\mathit{w}}_t-{\eta }_t{\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right).\end{array}$$

(12)

Case 2: For $F\left(\mathit{w}\right)={\sum }_{i=1}^N{\mathit{w}}_i\ln \left({\mathit{w}}_i\right)-{\mathit{w}}_i$ and corresponding (componentwise) link function $f\left(\mathit{w}\right)=\ln \left(\mathit{w}\right)$ we obtain a (multiplicative) Exponentiated Gradient (EG) update also called the unnormalized EG update (or EGU) [20]:

$$\begin{array}{c}\hfill {\displaystyle \frac{d\ln \hspace{0.33em}\left(\mathit{w}\left(t\right)\right)}{dt}}=-\mu {\nabla }_{\mathit{w}}L\left(\mathit{w}\left(t\right)\right),\hspace{0.33em}\mathit{w}\left(t\right)>0\hspace{0.33em}\hspace{0.33em}\forall \hspace{0.33em}t.\end{array}$$

(13)

In this sense, the unnormalized exponentiated gradient update (EGU) corresponds to the discrete-time version of the continuous ODE, obtained via Euler’s rule:

$$\begin{array}{cc}\hfill {\mathit{w}}_{t+1}& =\exp \left(\ln \left({\mathit{w}}_t\right)-\mu \Delta t\hspace{0.33em}{\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right)\\ & ={\mathit{w}}_t\odot \exp \left(-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right),\end{array}$$

where ⊙ and exp are componentwise multiplication and componentwise exponentiation, respectively, and $\eta =\mu \Delta t>0$ is the learning for discrete-time updates. This multiplicative update naturally preserves positivity constraints and corresponds to the natural geometry of the probability simplex.

2.6. Motivation for Using Parameterized Deformed Logarithms

Traditional Mirror Descent methods suffer from geometric rigidity—the fixed choice of mirror map f cannot adapt to diverse problem structures or data distributions. This limitation motivates our investigation of parameterized mirror maps based on trace-form entropies.

Adaptive Geometric Framework: Our approach addresses this fundamental limitation by introducing hyperparameter-controlled mirror maps $f_{\Theta }\left(\mathit{w}\right)$ that can:

• Adapt to statistical properties of training distributions.
• Interpolate between different geometries (e.g., Euclidean, exponential family, power-law).
• Provide automatic regularization through geometric bias.
• Enable systematic geometry exploration rather than ad-hoc selection.

Information-Theoretic Foundation: The connection between exponential families and Bregman divergences suggest that optimal mirror maps should reflect the underlying statistical structure of optimization problems. In fact, trace-form entropies provide systematic frameworks for discovering these optimal geometric structures.

There are many potential choices of mirror map $f\left(\mathit{w}\right)$ that can model the geometry of various optimization problems and adapt to the distribution of training data. In high dimensions (large-scale optimization), it can be advantageous to abandon the Euclidean geometry to improve convergence rates and performance. Using mirror descent with an appropriately chosen function we can obtain a considerable improvement.

3. Why Trace Entropies and Deformed Logarithms in MD and GEG?

Entropy measures provide natural regularization mechanisms and geometric structures for optimization algorithms. The connection between entropies, information theory, and geometry runs deep; each entropy functional induces a deformed logarithm and a unique Riemannian manifold structure through its associated Fisher information metric [32,33,34,35].

Trace entropies are functionals expressible in the explicit summation form [36,37,38,39,40,41]

$$\begin{array}{c}\hfill S\left(\mathit{p}\right)=\sum _i{p}_{i}f(1/p_i),\end{array}$$

(14)

where $p_i$ are probability values and $f(·)$ is a suitable monotonically increasing function. The term “trace” refers to their mathematical structure, which resembles the trace operation for matrices, i.e., a direct summation over individual components.

Trace entropies are intimately connected to deformed logarithms through $f\left(x\right)={\log }_D\left(x\right)$, where ${\log }_D$ represents a deformed logarithm function with specific mathematical properties ensuring proper entropic behavior [34,38].

A function ${\log }_D\left(x\right)$ qualifies as a deformed logarithm if it satisfies the following conditions:

• Domain ${\log }_D\left(x\right)$: ${\mathbb{R}}^{+}\to \mathbb{R}$
• Strictly monotonically increasing: ${\displaystyle \frac{d{\log }_D\left(x\right)}{dx}>0}$
• Concavity (optional): ${\displaystyle \frac{d^2{\log }_D\left(x\right)}{d{x}^2}<0}$
• Scaling and normalization: ${\log }_D\left(1\right)=0$, ${\displaystyle \frac{d{\log }_D\left(x\right)}{dx}{|}_{x=1}=1}$
• Duality: ${\log }_D(1/x)=-{\log }_{\tilde{D}}\left(x\right)$.

These axioms ensure that deformed logarithms generate well-behaved entropy functionals while providing sufficient mathematical flexibility for geometric adaptation. The concavity requirement ensures that resulting entropies satisfy the maximum entropy principle, while duality guarantees symmetric treatment of probabilities and their reciprocals, which are essential for consistent statistical interpretation.

Remark 1. 

It should be noted that since the generating (potential) function $F\left(\mathit{w}\right)$, (which is the integral of the link function $f\left(\mathit{w}\right)$) must be strictly convex, it is sufficient that the link function $f\left(\mathit{w}\right)={\log }_D\left(\mathit{w}\right)$ is a strictly monotonically increasing function, i.e., its first derivative must be positive but the negativity of its second derivative is in the general not necessary.

It should be noted that deformed logarithms and their corresponding deformed exponential functions can be flexibly tuned by one or more hyperparameters, whose optimization enables the adaptation to specific data distributions and problem geometries. By tuning/learning these hyperparameters, we adapt to the distribution of training data and/or we can adjust them to achieve desired properties of gradient descent algorithms.

It is of great importance to understand the mathematical structure of the generalized logarithms and its inverse generalized exponentials in order to obtain more insight into the proposed MD or EG update schemes. Motivated by this fact, and to make this paper more self-contained, we systematically revise fundamental properties of the deformed logarithms and their inverses, generalized exponentials and investigate links between them.

We provide in the Appendix A the basics of q-algebra and $\kappa$-algebra and calculus in Appendix A.1 [42].

4. MD and GEG Updates Using the Tsallis Entropy and Its Extensions

4.1. Properties of the Tsallis q-Logarithm and q-Exponential

In physics, the Tsallis entropy is a generalization of the standard Boltzmann–Gibbs entropy [34,43,44]. It is proportional to the expectation of the deformed q-logarithm (referred here as the Tsallis logarithm or termed logarithm) of a distribution.

The Tsallis q-logarithm is defined for $x>0$ as [45]

$$\begin{array}{c}\hfill {\log }_q^T\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{x^{\hspace{0.33em}1-q}-1}{1-q}}& \mathrm{for}\hspace{0.33em}\hspace{0.33em}x>0\hspace{0.33em}\hspace{0.33em}\mathrm{if}\hspace{0.33em}\hspace{0.33em}q\ne 1,\hfill \\ \\ \ln \left(x\right)& \mathrm{for}\hspace{0.33em}\hspace{0.33em}x>0\hspace{0.33em}\hspace{0.33em}\mathrm{if}\hspace{0.33em}\hspace{0.33em}q=1\hfill \\ \\ -\infty & \mathrm{for}\hspace{0.33em}\hspace{0.33em}x=0\hspace{0.33em}\hspace{0.33em}\forall q.\hfill \end{array}\right.\end{array}$$

(15)

The inverse function of the Tsallis q-logarithm is the deformed q-exponential function ${\exp }_q^T\left(x\right)$, defined as follows [45]:

$$\begin{array}{c}\hfill {\exp }_q^T\left(x\right)=\left\{\begin{array}{cc}{\displaystyle {[1+(1-q)x]}_{+}^{1/(1-q)}}& \mathrm{for}\left\{\begin{array}{c}x\in (-\infty ,1/(1-q\left)\right)\hspace{0.33em}\hspace{0.33em}\mathrm{if}\hspace{0.33em}\hspace{0.33em}q<1,\\ x\in (1/(q-1),\infty )\hspace{0.33em}\hspace{0.33em}\mathrm{if}\hspace{0.33em}\hspace{0.33em}q>1,\end{array}\right.\hfill \\ \\ \exp \left(x\right)& \mathrm{for}\hspace{0.33em}\hspace{0.33em}q=1.\hfill \end{array}\right.\end{array}$$

(16)

It is easy to check that these functions satisfy the following relationships:

$$\begin{array}{ccc}\hfill {\log }_q^T\left({\exp }_q^T\left(x\right)\right)& =& x,\hspace{0.33em}\hspace{0.33em}(0<{\exp }_q^T\left(x\right)<\infty ),\hfill \end{array}$$

(17)

$$\begin{array}{ccc}{\exp }_q^T\left({\log }_q^T\left(x\right)\right)\hfill & =& x,\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}x>0.\hfill \end{array}$$

(18)

Remark 2. 

The q-deformed exponential and logarithmic functions were introduced in Tsallis statistical physics in 1994 [44]. However, the q-deformation is related to the Box–Cox transformation (for $q=1-\lambda$), which was proposed in 1964 [46].

The plots of the q-logarithm and q-exponential functions for various values of q are illustrated in Figure 2 and Figure 3.

It should be noted that q-functions can be approximated by power series as follows:

$$\begin{array}{c}\hfill {\log }_q^T\left(x\right)\approx \ln \left(x\right)+{\displaystyle \frac{1}{2}}\left(\right|1-{q\left|\right)\left(\ln \left(x\right)\right)}^2+{\displaystyle \frac{1}{6}}{(1-q)}^2{\left(\ln \left(x\right)\right)}^3+\cdots ,\end{array}$$

(19)

and

$$\begin{array}{ccc}{\exp }_q^T\left(x\right)\hfill & \approx & 1+x+{\displaystyle \frac{1}{2}}q{x}^2+{\displaystyle \frac{1}{6}}\left(2q^2-q\right)x^3\cdots \hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & =\exp \left(x\right)+{\displaystyle \frac{1}{2}}\left(q-1\right)x^2+{\displaystyle \frac{1}{6}}\left(2q^2-q-1\right)x^3+O\left(x^4\right).\hfill \end{array}$$

(21)

These functions have the following basic properties [44,47,48,49]:

$$\begin{array}{ccc}\hfill {\log }_q^T\left(x\right)& =& -{\log }_{2-q}^T(1/x)\hfill \end{array}$$

(22)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\mathsf{\partial }{\log }_q^T\left(x\right)}{\mathsf{\partial }x}}& =& {\displaystyle \frac{1}{x^q}}>0\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{{\mathsf{\partial }}^2{\log }_q^T\left(x\right)}{\mathsf{\partial }{x}^2}}& =& {\displaystyle \frac{-q}{x^{q+1}}}<0\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}q>0.\hfill \end{array}$$

(24)

It is easy to prove the following fundamental properties:

$$\begin{array}{c}\hfill {\log }_q^T\left(xy\right)={\log }_q^T\left(x\right)+{\log }_q^T\left(y\right)+(1-q){\log }_q^T\left(x\right)){\log }_q^T\left(y\right)\hspace{0.33em}\hspace{0.33em}\mathrm{if}\hspace{0.33em}\hspace{0.33em}x>0,y>0\end{array}$$

(25)

$$\begin{array}{c}\hfill {\exp }_q^T\left(x\right){\exp }_q^T\left(y\right)={\exp }_q^T(x+y+(1-q)xy).\end{array}$$

(26)

Using these properties we can define nonlinear generalized algebraic forms q-sum and the q-product (for more details about q-algebra, see [47,50]

$$\begin{array}{cccc}\hfill x{\oplus }_q^{T}y& =& x+y+(1-q)xy,\hspace{0.33em}(x{\oplus }_1^{T}y=x+y),\hfill & \end{array}$$

(27)

$$\begin{array}{ccc}\hfill x{\otimes }_{q}y& =& {[x^{1-q}+y^{1-q}-1]}_{+}^{1/(1-q)}\hspace{0.33em}\hspace{0.33em}\mathrm{if}\hspace{0.33em}\hspace{0.33em}x>0,\hspace{0.33em}\hspace{0.33em}y>0\hspace{0.33em}(x{\otimes }_1^{T}y=x\hspace{0.33em}y).\hfill \end{array}$$

(28)

Using this notation, and definitions of q-exponential function (16) and q-logarithm (15), we can write the following formulas

$$\begin{array}{c}\hfill {\exp }_q^T(x+y)={\exp }_q^T\left(x\right){\otimes }_q{\exp }_q^T\left(y\right),\\ \hfill \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}1+(1-q)x>0,1+(1-q)y>0,1+(1-q)(x+y)>0,\end{array}$$

(29)

$$\begin{array}{c}\hfill {\exp }_q^T({\log }_q^T\left(x\right)+y)=x{\otimes }_q{\exp }_q^T\left(y\right),\\ \hfill \mathrm{for}x>0,1+(1-q)y>0,x^{1-q}+(1-q)y>0.\end{array}$$

(30)

which play a key role in this paper.

4.2. MD and GEG Updates Using the Tsallis q-Logarithm

Let us assume that the link function in Mirror Descent can take the following componentwise form

$$\begin{array}{c}\hfill f_q\left(\mathit{w}\right)={\log }_q^T\left(\mathit{w}\right),\hspace{0.33em}\mathit{w}={[{\mathit{w}}_1,\dots ,{\mathit{w}}_N]}^T\in {\mathbb{R}}_{+}^N.\end{array}$$

(31)

In this case the generating function $F\left(\mathit{w}\right)={\sum }_i{\mathit{w}}_i{\log }_q\left({\mathit{w}}_i\right)-{\log }_{q-1}\left({\mathit{w}}_i\right)$ and the Bregman divergence is a well-known beta divergence [28]:

$$\begin{array}{ccc}\hfill D_{F_q}({\mathit{w}}_{t+1}\left|\right|{\mathit{w}}_t)& =& \sum _{i=1}^N{\mathit{w}}_{i,t+1}\left({\log }_q\left({\mathit{w}}_{i,t+1}\right)-{\log }_q\left({\mathit{w}}_{i,t}\right)\right)-{\log }_{q-1}\left({\mathit{w}}_{i,t+1}\right)+{\log }_{q-1}\left({\mathit{w}}_{i,t}\right)\hfill \\ & =& \sum _{i=1}^N{\mathit{w}}_{i,t+1}{\displaystyle \frac{{\mathit{w}}_{i,t+1}^{1-q}-{\mathit{w}}_{i,t}^{1-q}}{1-q}}-{\displaystyle \frac{{\mathit{w}}_{i,t+1}^{2-q}-{\mathit{w}}_{i,t}^{2-q}}{2-q}},\hfill \end{array}$$

(32)

where $\beta =1-q,\hspace{0.33em}\hspace{0.33em}\beta \ne 0$.

Applying Equation (6) and taking into account Formula (31), we obtain a novel generalized exponentiated gradient update referred to as q-GEG or q-MD update

$$\boxed{{\mathit{w}}_{\mathit{t}+1}={\exp }_q^T\left[{\log }_q^T({\mathit{w}}_t)-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right]={\mathit{w}}_t{\otimes }_q{\exp }_q^T\left(-\eta {\nabla }_{\mathit{w}}L(\right.{\mathit{w}}_t\left)\right)}$$

(33)

where ${\otimes }_q$q-product defined by Equation (28) is performed componentwise.

The above q-MD update can be written in a scalar (componentwise) form as

$$\boxed{\begin{array}{c}\hfill {\mathit{w}}_{i,t+1}={\mathit{w}}_{i,t}{\otimes }_q{\exp }_q^T\left(-\eta {\nabla }_{{\mathit{w}}_i}L(\right.{\mathit{w}}_t\left)\right)\\ \hfill =\left[{\mathit{w}}_{i,t}^{1-q}+\left({\exp }_q^T(-\eta {\nabla }_{{\mathit{w}}_i}L(\right.\right.{\mathit{w}}_t)){)}^{1-q}-1{]}_{+}^{1/1-q}\end{array}}$$

(34)

By applying the property (31) and substituting $y\to y/(1-(1-q\left)x\right)$, we can obtain the following identity

$$\begin{array}{c}\hfill {\exp }_q^T(x+y)={\exp }_q^T\left(x\right)\hspace{0.33em}{\exp }_q^T\left({\displaystyle \frac{y}{1+(1-q)x}}\right)\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\left\{\begin{array}{c}x,y\in (-\infty ,1/(1-q\left)\right)\hspace{0.33em}\hspace{0.33em}\mathrm{if}\hspace{0.33em}\hspace{0.33em}q<1,\\ x,y\in (1/(q-1),\infty )\hspace{0.33em}\hspace{0.33em}\mathrm{if}\hspace{0.33em}\hspace{0.33em}q>1,\end{array}\right.\end{array}$$

(35)

Hence, we obtain a simplified generalized q-GEG update

$$\boxed{{\mathit{w}}_{i,t+1}={\mathit{w}}_{i,t}{\exp }_q^T\left({\displaystyle \frac{-\eta {\nabla }_{{\mathit{w}}_i}L\left({\mathit{w}}_t\right)}{{\mathit{w}}_{i,t}^{1-q}}}\right),}$$

(36)

which can be written in a compact vector form

$$\boxed{{\mathit{w}}_{t+1}={\mathit{w}}_t\odot {\exp }_q^T\left[-{\eta }_t\odot {\nabla }_{\mathit{w}}L({\mathit{w}}_t)\right]}$$

(37)

where a vector of learning rates

$${\eta }_t={[{\eta }_{1,t},\dots ,{\eta }_{N,t}]}^T,$$

has entries ${\eta }_{i,t}=\eta /\left((1+(1-q){\log }_q^T\left({\mathit{w}}_{i,t}\right)\right)=\eta \hspace{0.33em}{\mathit{w}}_{i,t}^{q-1}$.

Remark 3. 

Assuming that learning rate is time variable and it is represented by a vector, i.e., $\mathit{\eta }\to \eta {\mathit{w}}_t^{1-\alpha }$, where ${\eta }_t=\eta {\mathit{w}}_t^{1-\alpha -\beta }$, the proposed MD update takes particular form derived and extensively experimentally tested in our recent publication [15], however using a different approach employing alpha-beta divergences [14,27].

4.3. MD and EG Using Schwämmle–Tsallis (ST) Entropy

Schwämmle and Tsallis proposed two-parameter entropy [51]

$$\begin{array}{c}\hfill S_{q,q^{\prime }}^{ST}\left(p\right)=\sum _{i=1}^{\mathit{w}}{p}_i{\log }_{q,q^{\prime }}^{ST}(1/p_i),\hspace{0.33em}\hspace{0.33em}q\ne 1,\hspace{0.33em}\hspace{0.33em}{q}^{\prime }\ne 1,\end{array}$$

(38)

where the deformed logarithm referred to as the ST-logarithm or ST $(q,q^{\prime })$-logarithm is defined as

$$\begin{array}{c}\hfill {\log }_{q,q^{\prime }}^{ST}\left(x\right)={\log }_{q^{\prime }}^T\left({\left[x\right]}_q\right)={\log }_{q^{\prime }}^T{e}^{{\log }_q^T\left(x\right)}={\displaystyle \frac{1}{1-q^{\prime }}}\left[\exp \left({\displaystyle \frac{1-q^{\prime }}{1-q}}(x^{1-q}-1)\right)-1\right]\end{array}$$

(39)

for $x>0$ and $q\ne q^{\prime }$ (typically $q>1$ and $q^{\prime }<1$), where ${\left[x\right]}_q=\exp \left(lo{g}_q\left(x\right)\right)$ and its inverse function is formulated as a two-parameter deformed exponential, ${\exp }_{q,q^{\prime }}\left(x\right)$

$$\begin{array}{c}\hfill {\exp }_{q,q^{\prime }}^{ST}\left(x\right)={\left[1+{\displaystyle \frac{1-q}{1-q^{\prime }}}\ln (1+(1-q^{\prime })x)\right]}^{1/(1-q)}.\end{array}$$

(40)

Note that if either parameter q or $q^{\prime }$, or both, take the value of one, the above functions simplifies to Tsallis q-functions (logarithm and exponential), so we can write

$$\begin{array}{c}\hfill {\log }_{q,1}^{ST}\left(x\right)={\log }_{1,q}^{ST}\left(x\right)={\log }_q^T\left(x\right),\hspace{0.33em}{\log }_{1,1}^{ST}\left(x\right)=\ln \left(x\right)\end{array}$$

(41)

$$\begin{array}{c}\hfill {\exp }_{q,1}^{ST}\left(x\right)={\exp }_{1,q}^{ST}\left(x\right)={\exp }_q^T\left(x\right),\hspace{0.33em}\hspace{0.33em}{\exp }_{1,1}^{ST}\left(x\right)=\exp \left(x\right).\end{array}$$

(42)

In the special case for $q=q^{\prime }$, we obtain

$$\begin{array}{ccc}\hfill {\log }_{q,q}^{ST}\left(x\right)& =& {\displaystyle \frac{1}{1-q}}\left[\exp (x^{1-q}-1)-1\right]\hfill \\ & =& {\displaystyle \frac{1}{1-q}}\left[\exp \left((1-q){\log }_q^T\left(x\right)\right)-1\right],\hspace{0.33em}x>0,\hspace{0.33em}\hspace{0.33em}q>0\hfill \end{array}$$

(43)

and

$$\begin{array}{c}\hfill {\exp }_{q,q}^{ST}\left(x\right)={\left[\ln \left(x(1-q)+1\right)+1\right]}_{+}^{1/(1-q)}.\end{array}$$

(44)

The plots of the $(q,q^{\prime })$-logarithm and $(q,q^{\prime })$-exponential for various values of $q=q^{\prime }$ are illustrated in Figure 4.

Moreover, it is easy to prove the following useful properties

$$\begin{array}{c}\hfill {\log }_{q,q^{\prime }}^{ST}(1/x)=-{\log }_{2-q,2-q^{\prime }}^{ST}\left(x\right),\end{array}$$

(45)

$$\begin{array}{c}\hfill {\displaystyle \frac{d{\log }_{q.q^{\prime }}^{ST}\left(x\right)}{dx}}=x^{-q}\exp \left({\displaystyle \frac{1-q^{\prime }}{1-q}}(x^{1-q}-1)\right)>0\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}x>0,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\forall q,q^{\prime },\hspace{0.33em}\hspace{0.33em}q\ne 1.\end{array}$$

(46)

Defining $(q,q^{\prime })$-product as [52]:

$$\begin{array}{c}\hfill x{\otimes }_{q,q^{\prime }}^{ST}y={\exp }_{q{q}^{\prime }}\left({\log }_{q,q^{\prime }}\left(x\right)+{\log }_{q,q^{\prime }}\left(y\right)\right)\end{array}$$

(47)

we have the key formulas

$$\begin{array}{ccc}{\exp }_{q,q^{\prime }}^{ST}(x+y)\hfill & =& \hfill {\exp }_{q,q^{\prime }}^{ST}\left(x\right)\hspace{0.33em}\hspace{0.33em}{\otimes }_{q,q^{\prime }}\hspace{0.33em}\hspace{0.33em}{\exp }_{q,q^{\prime }}^{ST}\left(y\right),\end{array}$$

(48)

$$\begin{array}{ccc}\hfill {\exp }_{q,q^{\prime }}^{ST}({\log }_{q,q^{\prime }}^{ST}\left(x\right)+y)& =& x\hspace{0.33em}\hspace{0.33em}{\otimes }_{q,q^{\prime }}\hspace{0.33em}\hspace{0.33em}{\exp }_{q,q^{\prime }}^{ST}\left(y\right).\hfill \end{array}$$

(49)

Let us consider now that the link function is defined componentwise as

$$\begin{array}{c}\hfill f_{q,q^{\prime }}\left(\mathit{w}\right)={\log }_{q,q^{\prime }}^{ST}\left(\mathit{w}\right).\end{array}$$

(50)

The novel $(q,q^{\prime })$-GEG update can take the following form

$$\boxed{{\mathit{w}}_{t+1}={\mathit{w}}_t{\otimes }_{q,q^{\prime }}{\exp }_{q,q^{\prime }}^{\mathrm{ST}}\left(-\eta {\nabla }_{\mathit{w}}L(\right.{\mathit{w}}_t\left)\right),}$$

(51)

In this case, the update is more complex than in the previous case. An alternative approach is to apply the MMD/NG Formula (10):

$$\boxed{{\displaystyle {\mathit{w}}_{t+1}=[{\mathit{w}}_t-\eta \mathrm{diag}\{{\mathit{w}}^q\odot \exp \left(\frac{1-q^{\prime }}{1-q}(1-\right.{\mathit{w}}^{1-q}\left)\right\}{\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right){]}_{+}}}$$

(52)

which can be written equivalently in a scalar form as

$$\boxed{{\mathit{w}}_{i,t+1}=\left[{\mathit{w}}_{i,t}-\eta {\mathit{w}}_{i,t}^q\exp \left({\displaystyle \frac{1-q^{\prime }}{1-q}}(1-\right.\right.{\mathit{w}}_{i,t}^{1-q}){\displaystyle \frac{\mathsf{\partial }L\left({\mathit{w}}_t\right)}{\mathsf{\partial }{\mathit{w}}_i}}{]}_{+}}$$

(53)

Remark 4. 

Extension to Three Parameters $(q,q^{\prime },r)$-Logarithm. Note that by using definition ${\left[x\right]}_q=\exp \left({\log }_q^T\left(x\right)\right)$ we can write the ST logarithm in a compact form

$$\begin{array}{c}\hfill {\log }_{q,q^{\prime }}^{ST}\left(x\right)={\displaystyle \frac{{\left[x\right]}_q^{1-q^{\prime }}-1}{1-q^{\prime }}}={\displaystyle \frac{[\exp {({\log }_q^T\left(x\right)]}^{1-q^{\prime }}-1}{1-q^{\prime }}}\end{array}$$

(54)

Analogously, we can define

$$\begin{array}{c}\hfill {\left[x\right]}_{q,q^{\prime }}=\exp \left({\log }_{q,q^{\prime }}^{ST}\left(x\right)\right).\end{array}$$

(55)

Hence, we can formulate a three-parameter logarithm as proposed in [53]

$$\begin{array}{c}\hfill {\log }_{q,q^{\prime },r}^{CC}\left(x\right)={\displaystyle \frac{{\left({\left[x\right]}_{q,q^{\prime }}^{1-q^{\prime }}\right)}^{1-r}-1}{1-r}}={\displaystyle \frac{(\exp {\left({\log }_{q,q^{\prime }}^{ST}\left(x\right)\right)}^{1-r}-1}{1-r}}\end{array}$$

(56)

The plots of the $(q,q^{\prime },r)$-logarithm and $(q,q^{\prime },r)$-exponential for coincident values of q, $q^{\prime }$, and r are illustrated in Figure 5.

In a similar way, as in the previous section, we can drive MD updates using as a link function the above-defined three-parameter logarithm.

5. MD and GEG Using the Kaniadakis Entropy and Its Extensions and Generalizations

5.1. Basic Properties of $\kappa$-Logarithm and $\kappa$-Exponential

An entropic structure emerging in the context of special relativity, is the one defined by Kaniadakis [36,37] as follows

$$\begin{array}{c}\hfill S_{\kappa }\left(p_i\right)=\sum _i{p}_i{\log }_{\kappa }^K(1/p_i),\end{array}$$

(57)

where a deformed $\kappa$-logarithm referred to as the Kaniadakis $\kappa$-logarithm is defined as [36,37]:

$$\begin{array}{c}\hfill {\log }_{\kappa }^K\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{x^{\kappa }-x^{-\kappa }}{2\kappa }=\frac{1}{\kappa }\sinh \left(\kappa \ln \left(x\right)\right)}& \mathrm{if}\hspace{0.33em}\hspace{0.33em}x>0\hspace{0.33em}\hspace{0.33em}\mathrm{and}\hspace{0.33em}\hspace{0.33em}0<{\kappa }^2<1\hfill \\ \\ \ln \left(x\right)& \mathrm{if}\hspace{0.33em}\hspace{0.33em}x>0\hspace{0.33em}\hspace{0.33em}\mathrm{and}\hspace{0.33em}\hspace{0.33em}\kappa =0.\hfill \end{array}\right.\end{array}$$

(58)

The inverse function of the Kaniadakis $\kappa$-logarithm is the deformed exponential function ${\exp }_{\kappa }^K\left(x\right)$, represented as

$$\begin{array}{ccc}\hfill {\exp }_{\kappa }^K\left(x\right)& =& \exp \left({\mathsf{\int }}_0^x{\displaystyle \frac{dy}{\sqrt{1+{\kappa }^2{y}^2}}}\right)\hfill \\ & =& \left\{\begin{array}{cc}{\displaystyle {\left(\sqrt{1+{\kappa }^2{x}^2}+\kappa x\right)}^{1/\kappa }=\exp \left(\frac{1}{\kappa }\hspace{0.33em}\mathrm{arsinh}\left(\kappa x\right)\right)}& -1<\kappa <1\hfill \\ \\ \exp \left(x\right)& \kappa =0.\hfill \end{array}\right.\hfill \end{array}$$

(59)

The plots of the $\kappa$-logarithm and $\kappa$-exponential functions for various values of $\kappa$ are illustrated in Figure 6 and Figure 7.

Note that the Kaniadakis logarithm can also be expressed in terms of the Tsallis logarithm as

$$\begin{array}{c}\hfill {\log }_{\kappa }^K\left(x\right)={\displaystyle \frac{{\log }_{1+\kappa }^T\left(x\right)+{\log }_{1-\kappa }^T\left(x\right)}{2}}.\end{array}$$

(60)

These functions have the following fundamental and useful properties [37,41]:

$$\begin{array}{ccc} {\log }_{\kappa }^K\left(1\right)\hfill & =& 1,\hspace{0.33em}{\log }_{\kappa }^K\left(0^{+}\right)=-\infty ,\hspace{0.33em}{\log }_{\kappa }^K(\infty )=+\infty ,\hfill \end{array}$$

(61)

$$\begin{array}{ccc}{\log }_{\kappa }^K(1/x)\hfill & =& -{\log }_{\kappa }^K\left(x\right),\hfill \end{array}$$

(62)

$$\begin{array}{ccc} {\log }_{\kappa }^K\left(x^{\lambda }\right)\hfill & =& \lambda \hspace{0.33em}{\log }_{\lambda \hspace{0.33em}\kappa }^K\left(x\right),\hfill \end{array}$$

(63)

$$\begin{array}{ccc} {\log }_{\kappa }^K(x\hspace{0.33em}y)\hfill & =& y^{\kappa }{\log }_{\kappa }^K\left(x\right)+x^{-\kappa }{\log }_{\kappa }^K\left(y\right),\hfill \end{array}$$

(64)

$$\begin{array}{ccc}{\log }_{\kappa }\left(\exp \left(x\right)\right)\hfill & =& (1/\kappa )\sinh \left(\kappa x\right),\hfill \end{array}$$

(65)

$$\begin{array}{ccc} \ln \left({\exp }_{\kappa }\left(x\right)\right)\hfill & =& (1/\kappa )\mathrm{arsinh}\left(\kappa x\right),\hfill \end{array}$$

(66)

$$\begin{array}{ccc} {\displaystyle \frac{\mathsf{\partial }{\log }_{\kappa }^K\left(x\right)}{\mathsf{\partial }x}}\hfill & =& {\displaystyle \frac{x^{\kappa }+x^{-\kappa }}{2x}}={\displaystyle \frac{\cosh \left[\kappa \ln \right(x\left)\right]}{x}}>0,\hfill \end{array}$$

(67)

$$\begin{array}{ccc} {\displaystyle \frac{{\mathsf{\partial }}^2{\log }_{\kappa }^K\left(x\right)}{\mathsf{\partial }{x}^2}}\hfill & =& {\displaystyle \frac{\kappa -1}{2}}{x}^{\kappa -2}-{\displaystyle \frac{\kappa +1}{2}}{x}^{-\kappa -2}<0\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}\kappa \in [-1,1].\hfill \end{array}$$

(68)

The last two properties indicate that the Kaniadakis $\kappa$-logarithm is monotonically increasing for any value of $\kappa$, and for $\left|\kappa \right|<1,$ it is additionally a strictly concave function.

The $\kappa$-logarithm can be approximated as a power series

$$\begin{array}{c}\hfill {\log }_{\kappa }\left(x\right)\approx \ln \left(x\right)+{\displaystyle \frac{1}{3!}}{\kappa }^2{\left[\ln \left(x\right)\right]}^3+{\displaystyle \frac{1}{5!}}{\kappa }^4{\left[\ln \left(x\right)\right]}^5+{\displaystyle \frac{1}{7!}}{\kappa }^6{\left[\ln \left(x\right)\right]}^7\cdots ,\hspace{0.33em}\hspace{0.33em}x>0.\end{array}$$

(69)

Furthermore, it is important to note that applying the Taylor series expansion of the $\kappa$-exponential we can obtain a simple approximation as

$$\begin{array}{ccc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\exp }_{\kappa }^K\left(x\right)& =& 1+x+{\displaystyle \frac{1}{2!}}{x}^2+{\displaystyle \frac{1}{3!}}(1-{\kappa }^2)x^3+{\displaystyle \frac{1}{4!}}(1-4{\kappa }^4)x^4+\cdots \hfill \end{array}$$

(70)

$$\begin{array}{cc}& =\exp \left(x\right)-{\displaystyle \frac{1}{3!}}{\kappa }^2{x}^3-{\displaystyle \frac{1}{4!}}4{\kappa }^4{x}^4+\cdots .\hfill \end{array}$$

(71)

Two notable features of the $\kappa$-exponential function are that it asymptotically approaches a regular exponential function for small x and asymptotically approaches a power law for large value $\left|x\right|$ [37,54]. Specifically,

$$\begin{array}{c}\hfill \underset{x\to 0}{\lim }{\exp }_{\kappa }\left(x\right)\sim \hspace{0.33em}\exp \left(x\right),\end{array}$$

(72)

$$\begin{array}{c}\hfill \underset{x\to \pm \infty }{\lim }{\exp }_{\kappa }\left(x\right)\sim \hspace{0.33em}{\left|2\kappa x\right|}^{\pm 1/\left|\kappa \right|}.\end{array}$$

(73)

The $\kappa$-exponential function has the following basic properties [36,37,41]

$$\begin{array}{ccc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\exp }_{\kappa }^K\left(0\right)& =& 1,\hspace{0.33em}\hspace{0.33em}{\exp }_{\kappa }^K(-\infty )=0^{+},\hspace{0.33em}\hspace{0.33em}{\exp }_{\kappa }^K(\infty )=+\infty ,\hfill \end{array}$$

(74)

$$\begin{array}{ccc}\hfill {\exp }_{\kappa }^K(-\infty )& =& 0^{+},\hspace{0.33em}\hspace{0.33em}{\exp }_{\kappa }^K(+\infty )=+\infty ,\hfill \end{array}$$

(75)

$$\begin{array}{ccc}\hfill {\exp }_{\kappa }^K\left(x\right){\exp }_{\kappa }^K(-x)& =& 1,\hfill \end{array}$$

(76)

$$\begin{array}{ccc}\hfill {\left[{\exp }_{\kappa }^K\left(x\right)\right]}^r& =& {\exp }_{\kappa /r}^K\left(rx\right),\hspace{0.33em}\hspace{0.33em}r\in \mathbb{R},\hfill \end{array}$$

(77)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\mathsf{\partial }{\exp }_{\kappa }^K\left(x\right)}{\mathsf{\partial }x}}& >& 0,\hfill \end{array}$$

(78)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{{\mathsf{\partial }}^2{\exp }_{\kappa }^K\left(x\right)}{\mathsf{\partial }{x}^2}}& >0& \hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}\kappa \in [-1,1].\hfill \end{array}$$

(79)

The last two properties mean that the Kaniadakis $\kappa$-exponential is a monotonically increasing and convex function for a specific range of the parameter $\kappa$.

The property (79) emerges as a particular case of the more general one

$$\begin{array}{ccc}\hfill {\exp }_{\kappa }^K\left(x\right){\exp }_{\kappa }^K\left(y\right)& =& {\exp }_{\kappa }^K\left(x\right){\oplus }_{\kappa }{\exp }_{\kappa }^K\left(y\right),\hfill \end{array}$$

(80)

$$\begin{array}{ccc}\hfill {\log }_{\kappa }^K\left(xy\right)& =& {\log }_{\kappa }^K\left(x\right){\oplus }_{\kappa }{\log }_{\kappa }^K\left(y\right),\hfill \end{array}$$

(81)

where $\kappa$-addition is defined as

$$\begin{array}{ccc}\hfill x{\oplus }_{\kappa }y& =& x\hspace{0.33em}\sqrt{1+{\kappa }^2{y}^2}+y\hspace{0.33em}\sqrt{1+{\kappa }^2{x}^2}\hfill \end{array}$$

(82)

$$\begin{array}{cc} \hfill & \approx x+y+{\displaystyle \frac{{\kappa }^2}{2}}(x{y}^2+x^{2}y)-{\displaystyle \frac{{\kappa }^4}{8}}(x{y}^4+x^{4}y)\cdots \hfill \end{array}$$

(83)

By defining and evaluating the $\kappa$-product

$$\begin{array}{ccc}\hfill {\displaystyle x{\otimes }_{\kappa }y}& =& {\exp }_{\kappa }\left[{\log }_{\kappa }\left(x\right)+{\log }_{\kappa }\left(y\right)\right]\hfill \end{array}$$

(84)

$$\begin{array}{cc}\hfill & ={\left({\displaystyle \frac{x^{\kappa }-x^{-\kappa }}{2}}+{\displaystyle \frac{y^{\kappa }-y^{-\kappa }}{2}}+\hspace{0.33em}\sqrt{1+{\left({\displaystyle \frac{x^{\kappa }-x^{-\kappa }}{2}}+{\displaystyle \frac{y^{\kappa }-y^{-\kappa }}{2}}\right)}^2}\right)}^{1/\kappa }\hfill \end{array}$$

(85)

$$\begin{array}{cc} \hfill & =\exp \left({\displaystyle \frac{1}{\kappa }}\mathrm{arsinh}((x^{\kappa }-x^{-\kappa }+y^{\kappa }-y^{-\kappa })/2)\right)\hfill \end{array}$$

(86)

$$\begin{array}{cc} \hfill & ={\displaystyle \frac{1}{\kappa }}\sinh \left({\displaystyle \frac{1}{\kappa }}\mathrm{arsinh}\left(\kappa x\right)\hspace{0.33em}\mathrm{arsinh}\left(\kappa y\right)\right),\hfill \end{array}$$

(87)

we have the key formulas for our MD application

$$\begin{array}{c}\hfill {\exp }_{\kappa }^K(x+y)={\exp }_{\kappa }^K\left(x\right){\otimes }_{\kappa }{\exp }_{\kappa }^K\left(y\right),\end{array}$$

(88)

$$\begin{array}{c}\hfill {\exp }_{\kappa }^K({\log }_{\kappa }^K\left(x\right)+y)=x{\otimes }_{\kappa }{\exp }_{\kappa }^K\left(y\right).\end{array}$$

(89)

Appendix A.2 gives an overview of the $\kappa$-algebra and calculus.

5.2. MD and GEG Using the Kaniadakis Entropy and $\kappa$-Logarithm

Let us assume that the link function in Mirror Descent can take the following componentwise form

$$\begin{array}{c}\hfill f_{\kappa }\left({\mathit{w}}_t\right)={\log }_{\kappa }^K\left({\mathit{w}}_t\right),\hspace{0.33em}\mathit{w}={[{\mathit{w}}_{1,t},\dots ,{\mathit{w}}_{N,t}]}^T\in {\mathbb{R}}_{+}^N.\end{array}$$

(90)

Note that since the first derivative of ${\log }_{\kappa }^K\left(\mathit{w}\right)$ is positive and second derivative is negative, the link function is a (componentwise) increasing function and additionally a concave function for $\kappa \in (-1,1]$. In this case the generating function:

$$\begin{array}{c}\hfill F_{\kappa }\left({\mathit{w}}_t\right)=\sum _{i=1}^N{\displaystyle \frac{1}{2\kappa }}\left[{\displaystyle \frac{{\mathit{w}}_{i,t}^{1+\kappa }}{1+\kappa }}-{\displaystyle \frac{{\mathit{w}}_{i,t}^{1-\kappa }}{1-\kappa }}\right].\end{array}$$

(91)

Taking into account Formula (6), we obtain a novel $\kappa$-GEG update

$$\boxed{{\mathit{w}}_{t+1}={\mathit{w}}_t\hspace{0.33em}{\otimes }_{\kappa }\hspace{0.33em}{\exp }_{\kappa }^K\left(-\eta {\nabla }_{\mathit{w}}L(\right.{\mathit{w}}_t\left)\right)}$$

(92)

where ${\otimes }_{\kappa }$ is a $\kappa$-product defined by Equation (87) and is performed componentwise.

5.3. Two-Parameter Logarithms Based on Generalized Kaniadakis–Lissia–Scarfone Entropy

Another important generalized entropy has the following form

$$\begin{array}{c}\hfill S_{\kappa ,r}\left(p\right)=-\sum _{i=1}^{\mathit{w}}{\left(p_i\right)}^{r+1}{\displaystyle \frac{p_i^{\kappa }-p_i^{-\kappa }}{2\kappa }}=-\sum _{i=1}^{\mathit{w}}{p}_i{\log }_{\kappa ,r}\left(p_i\right),\end{array}$$

(93)

which was introduced by Sharma, Taneja and Mittal (STM) in [55,56,57], and also investigated, independently, by Kaniadakis, Lissia and Scarfone (KLS) in [38,39,58].

Equation (93) mimics the expression of the Boltzmann–Gibbs entropy by replacing the standard natural logarithm $\ln \left(x\right)$ with the two-parametric deformed logarithm ${\log }_{\kappa ,r}\left(x\right)$ defined as

$$\begin{array}{c}\hfill {\log }_{\kappa ,r}\left(x\right)=x^r\hspace{0.33em}{\displaystyle \frac{x^{\kappa }-x^{-\kappa }}{2\kappa }},\hspace{0.33em}\hspace{0.33em}x>0,\hspace{0.33em}\hspace{0.33em}r\in \mathbb{R},\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}-\left|\kappa \right|\le r\le 1/2-|1/2-|\kappa \left|\right|.\end{array}$$

(94)

The surface plots of the $(\kappa ,r)$-logarithm for various values of hyperparameters $\kappa$ and r are illustrated in Figure 8 and Figure 9.

The $(\kappa ,r)$-logarithm can be expressed via the Kaniadakis $\kappa$-logarithm and the Tsallis q-logarithm

$$\begin{array}{c}\hfill {\log }_{\kappa ,r}\left(x\right)=x^r\hspace{0.33em}{\displaystyle \frac{x^{\kappa }-x^{-\kappa }}{2\kappa }}=x^r\hspace{0.33em}{\log }_{\kappa }^K\left(x\right)=x^{r-\kappa }{\log }_{1-2\kappa }^T\left(x\right).\end{array}$$

(95)

Obviously, for $r=0$ the $(\kappa ,r)$-logarithm simplifies to a Kaniadakis $\kappa$-logarithm, and for $r=\pm \left|\kappa \right|$ one recovers the Tsallis q-logarithm with $q=1\mp 2\left|\kappa \right|$, ($0<q<2$).

By introducing a new parameter $\omega =r/\kappa$ or replacing $r=\omega \hspace{0.33em}\kappa$ we can represent the logarithm as

$$\begin{array}{c}\hfill {\log }_{\kappa ,\omega }\left(x\right)={\displaystyle \frac{x^{\kappa (\omega +1)}-x^{\kappa (\omega -1)}}{2\kappa }},\end{array}$$

(96)

which for $\omega =0$ simplifies to $\kappa$-logarithm and with $\omega =1$ and $\kappa =(1-q)/2$ we have a q-logarithm. This formula indicates that this logarithm smoothly interpolates between a Kaniadakis logarithm and Tsallis logarithm.

Summarizing, the $(\kappa ,r)$-logarithm can be described as follows [38,39]

$$\begin{array}{c}\hfill {\log }_{\kappa ,r}^{KLS}\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{x^{r+\kappa }-x^{r-\kappa }}{2\kappa }}& \mathrm{if}\hspace{0.33em}\hspace{0.33em}x>0\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}r\in \mathbb{R},\hspace{0.33em}\hspace{0.33em}\mathrm{and}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}-\left|\kappa \right|\le r\le \left|\kappa \right|,\hfill \\ \\ {\displaystyle {\log }_{\kappa }^K\left(x\right)=\frac{x^{\kappa }-x^{-\kappa }}{2\kappa }}& \mathrm{if}\hspace{0.33em}\hspace{0.33em}x>0\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}r=0,\hspace{0.33em}\kappa \in [-1,1],\hspace{0.33em}\hspace{0.33em}\kappa \ne 0,\hfill \\ \\ {\displaystyle {\log }_q^T\left(x\right)=\frac{x^{1-q}-1}{1-q},}& \mathrm{if}\hspace{0.33em}\hspace{0.33em}x>0\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}r=\kappa =(1-q)/2,\hspace{0.33em}\hspace{0.33em}q\ne 1,\hspace{0.33em}q>0,\hfill \\ \\ \ln \left(x\right)& \mathrm{if}\hspace{0.33em}\hspace{0.33em}x>0\hspace{0.33em}\hspace{0.33em}\mathrm{and}\hspace{0.33em}\hspace{0.33em}r=\kappa =0.\hfill \end{array}\right.\end{array}$$

(97)

It should also be noted the $(\kappa ,r)$-logarithm can be represented approximately by the following power series for relatively small $\kappa$:

$$\begin{array}{c}\hfill {\displaystyle {\log }_{\kappa ,r}^{KLS}\approx \ln \left(x\right)+\frac{1}{2}\left(2r\right){\left[\ln \left(x\right)\right]}^2+\frac{1}{6}({\kappa }^2+3r^2){\left[\ln \left(x\right)\right]}^3\cdots .}\end{array}$$

(98)

The $(\kappa ,r)$ logarithm has the following basic properties

$$\begin{array}{ccc}\hfill {\log }_{\kappa ,r}\left(1\right)& =& 0,\hspace{0.33em}{\log }_{\kappa ,r}\left(0^{+}\right)=-\infty ,\hspace{0.33em}\mathrm{for}\hspace{0.33em}r<\left|\kappa \right|,\hspace{0.33em}{\log }_{\kappa ,r}(+\infty )=+\infty ,\hspace{0.33em}\mathrm{for}\hspace{0.33em}r>-\left|\kappa \right|,\hfill \end{array}$$

$$\begin{array}{ccc}\hfill {\log }_{\kappa ,r}\left(x\right)& =& -{\log }_{\kappa ,-r}(1/x)={\log }_{-\kappa ,r}\left(x\right), \hfill \end{array}$$

(99)

$$\begin{array}{ccc}\hfill {\log }_{\kappa ,r}\left(x^{\lambda }\right)& =& \lambda \hspace{0.33em}{\log }_{\lambda \hspace{0.33em}\kappa }\left(x\right), \hfill \end{array}$$

(100)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\mathsf{\partial }{\log }_{\kappa ,r}\left(x\right)}{\mathsf{\partial }x}}& >0& ,\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}-\left|\kappa \right|\le r\le \left|\kappa \right|, \hfill \end{array}$$

(101)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{{\mathsf{\partial }}^2{\log }_{\kappa ,r}\left(x\right)}{\mathsf{\partial }{x}^2}}& <0& \hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}-\left|\kappa \right|\le r\le 1/2-|1/2-|\kappa \left|\right|. \hfill \end{array}$$

(102)

The last two properties indicate that the $(\kappa ,r)$-logarithm is a strictly monotonically increasing function for $-\left|\kappa \right|\le r\le \left|\kappa \right|$ and additionally a concave function for $-\left|\kappa \right|\le r\le 1/2-|1/2-|\kappa \left|\right|$.

Remark 5. 

Relation to the Euler Logarithm: It is interesting to note that for $r+k=a$ and $r-k=b$ the KLS $\kappa ,r$–logarithm can be represented as the Euler logarithm [17,59]:

$$\begin{array}{c}\hfill {\log }_{a,b}^{Eu}\left(x\right)={\displaystyle \frac{x^a-x^b}{a-b}},\hspace{0.33em}x>0,\hspace{0.33em}\hspace{0.33em}a<0,\hspace{0.33em}\hspace{0.33em}0<b<1,\end{array}$$

(103)

which is related to the Borges–Rodity entropy [50,60].

Connection to the Schwämmle–Tsallis logarithm: By applying nonlinear transformation in (103) $x\to \exp \left({\log }_q\left(x\right)\right)$ and $a=1-q^{\prime }$, $b=0$, we obtain the Schwämmle–Tsallis logarithm (39).

Connection to the Mean Value Theorem: The function has deep connections to the Mean Value Theorem applied to power functions. For the power function $g\left(t\right)=x^t$, the Mean Value Theorem guarantees the existence of some parameter $c\in (a,b)$ such that $g^{\prime }\left(c\right)={\displaystyle \frac{g\left(b\right)-g\left(a\right)}{b-a}}$, which yields:

$$\begin{array}{c}\hfill {\log }_{a,b}^{EU}\left(x\right)={\displaystyle \frac{x^b-x^a}{b-a}}=c·x^{c-1}·\ln \left(x\right)\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}c\in (a,b).\end{array}$$

(104)

Logarithmic Mean Connection: The function relates to the logarithmic mean $L(u,v)={\displaystyle \frac{u-v}{\ln u-\ln v}}$ through the substitution $u=x^a,v=x^b$. These connections provide alternative computational approaches and theoretical insights.

Exponential Function Theory: The underlying structure connects to exponential function differentiation rules, where ${\displaystyle \frac{d}{dt}}{x}^t=x^t\ln \left(x\right)$, explaining the limiting behavior observed in the analysis.

Computational and numerical considerations: The numerical analysis reveals several important computational aspects:

1. 

Numerical Stability: The KLS logarithm becomes increasingly stable as $x\to 1$, but exhibits potential numerical instability for x values far from unity.

2. 

Parameter Sensitivity: Small x values create higher sensitivity to parameter changes, requiring careful numerical handling.

3. 

Convergence Properties: The limiting behavior requires special computational treatment using L’Hôpital’s rule.

5.4. Exponential KLS Function and Its Properties

With the existence of ${\exp }_{\kappa ,r}\left(x\right)$, the inverse function of ${\log }_{\kappa ,r}\left(x\right)$ follows from its monotonicity in $\mathbb{R}$ although an explicit expression, in general, cannot be given. In other words, the inverse function can not be expressed in a closed analytical form but it can be approximated and expressed, for example, in terms of the Lambert–Tsallis $W_q$-functions, which are the solution of equations $W_q\left(z\right){[1+(1-q)W_q\left(z\right)]}_{+}^{1/(1-q)}=z$:

$$\begin{array}{c}\hfill {\displaystyle {\exp }_{\kappa ,r}\left(x\right)={\left(\frac{W_{\frac{\lambda +1}{\lambda }}\left(\lambda {\tilde{x}}^{-\lambda }\right)}{\lambda }\right)}^{-1/\left(2\kappa \right),}}\end{array}$$

(105)

where $\lambda =\left(2\kappa \right)/(r+\kappa )$, $\tilde{x}=2\kappa x$ and $W_q$ is the Lambert–Tsallis function [61].

Another much simpler approach is to use Lagrange’s inversion theorem around 1 to obtain the following rough power series approximation (which may be sufficient for most of our applications):

$$\begin{array}{ccc}\hfill {\displaystyle {\exp }_{\kappa ,r}\left(x\right)}& \approx & 1+x+{\displaystyle \frac{1}{2}}(1-2r)x^2+\left({\displaystyle \frac{1}{6}}-r+{\displaystyle \frac{3}{2}}{r}^2-{\displaystyle \frac{1}{6}}{\kappa }^2\right)x^3+\cdots \hfill \end{array}$$

(106)

$$\begin{array}{cc}\hspace{0.33em}\hfill & =\exp \left(x\right)-r{x}^2+\left({\displaystyle \frac{3}{2}}{r}^2-r-{\displaystyle \frac{1}{6}}{\kappa }^2\right)x^3+O\left(x^4\right).\hfill \end{array}$$

(107)

Hence, we can represent a $(\kappa ,r)$-exponential as follows:

$$\begin{array}{c}\hfill {\exp }_{\kappa ,r}\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \approx \exp \left(x\right)-r{x}^2+\left(\frac{3}{2}{r}^2-r-\frac{1}{6}{\kappa }^2\right)x^3}& \mathrm{for}\hspace{0.33em}r\in \mathbb{R},\hspace{0.33em}\hspace{0.33em}-\left|\kappa \right|\le r\le \left|\kappa \right|,\hspace{0.33em}\hspace{0.33em}\left|\kappa \right|<1,\hfill \\ \\ {\displaystyle {\exp }_{\kappa }^K\left(x\right)={\left[\kappa x+\sqrt{1+{\kappa }^2{x}^2}\right]}^{1/\kappa }}& \hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}r=0,\hspace{0.33em}\kappa \in [-1,1],\hspace{0.33em}\hspace{0.33em}\kappa \ne 0,\hfill \\ \\ {\displaystyle {\exp }_q^T\left(x\right)={\left[1+(1-q)x\right]}_{+}^{1/(1-q)}}& \mathrm{for}\hspace{0.33em}r=\kappa =(1-q)/2,\hspace{0.33em}\hspace{0.33em}q\ne 1,\hfill \\ \\ \exp \left(x\right)& \mathrm{for}\hspace{0.33em}\hspace{0.33em}r=\kappa =0.\hfill \end{array}\right.\end{array}$$

(108)

Furthermore, the $(\kappa ,r)$-exponential function has the following fundamental properties:

$$\begin{array}{ccc}\hfill {\exp }_{\kappa ,r}\left(0\right)& =& 1,\hspace{0.33em}\hspace{0.33em}{\exp }_{\kappa ,r}(-\infty )=0^{+},\hspace{0.33em}\mathrm{for}\hspace{0.33em}r<\left|\kappa \right|,\hspace{0.33em}{\exp }_{\kappa ,r}{(}_{\infty })=+\infty ,\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}r>-\left|\kappa \right|,\hfill \end{array}$$

$$\begin{array}{ccc}\hfill {\exp }_{\kappa ,r}\left(x\right){\exp }_{\kappa ,r}(-x)& =& 1, \hfill \end{array}$$

(109)

$$\begin{array}{ccc}\hfill {\left({\exp }_{\kappa ,r}\left(x\right)\right)}^{\lambda }& =& {\exp }_{\kappa /\lambda ,r/\lambda }\left(\lambda x\right), \hfill \end{array}$$

(110)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\mathsf{\partial }{\exp }_{\kappa ,r}\left(x\right)}{\mathsf{\partial }x}}& >& 0,\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}-\left|\kappa \right|\le r\le \left|\kappa \right|, \hfill \end{array}$$

(111)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{{\mathsf{\partial }}^2{\exp }_{\kappa ,r}\left(x\right)}{\mathsf{\partial }{x}^2}}& >& 0,\hspace{0.33em}\hspace{0.33em}\mathrm{for}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}-\left|\kappa \right|\le r\le 1/2-|1/2-|\kappa \left|\right| \hfill \end{array}$$

(112)

The last two properties mean that the $(\kappa ,r)$-exponential is a monotonically increasing convex function.

Two notable features of the $(\kappa ,r)$-logarithm and exponential function are that it asymptotically approaches a regular exponential function for small x and asymptotically approaches a power law for large absolute x:

$$\begin{array}{ccc}\hfill \underset{x\to 0^{+}}{\lim }{\log }_{\kappa ,r}\left(x\right)& \sim & \hspace{0.33em}{\displaystyle \frac{-1}{2\left|\kappa \right|\hspace{0.33em}{x}^{\left|\kappa \right|+r}}},\hfill \end{array}$$

(113)

$$\begin{array}{ccc}\hfill \underset{x\to +\infty }{\lim }{\log }_{\kappa ,r}\left(x\right)& \sim & \hspace{0.33em}{\displaystyle \frac{x^{\left|\kappa \right|+r}}{2\kappa }}, \hfill \end{array}$$

(114)

$$\begin{array}{ccc}\hfill \underset{x\to 0}{\lim }{\exp }_{\kappa ,r}\left(x\right)& \sim & \hspace{0.33em}\exp \left(x\right), \hfill \end{array}$$

(115)

$$\begin{array}{ccc}\hfill \underset{x\to \pm \infty }{\lim }{\exp }_{\kappa ,r}\left(x\right)& \sim & {\hspace{0.33em}\left|2\kappa x\right|}^{1/(r\pm |\kappa \left|\right)}.\hfill \end{array}$$

(116)

By defining

$$\begin{array}{c}\hfill {\displaystyle x{\otimes }_{\kappa ,r}y={\exp }_{\kappa ,r}\left[{\log }_{\kappa ,r}\left(x\right)+{\log }_{\kappa ,r}\left(y\right)\right]}\end{array}$$

(117)

we have the key formulas for our MD (GEG) implementations

$$\begin{array}{ccc}\hfill {\exp }_{\kappa ,r}(x+y)& =& {\exp }_{\kappa ,r}\left(x\right){\otimes }_{\kappa ,r}{\exp }_{\kappa ,r}^K\left(y\right),\hfill \end{array}$$

(118)

$$\begin{array}{ccc}\hfill {\exp }_{\kappa ,r}({\log }_{\kappa ,r}^K\left(x\right)+y)& =& x{\otimes }_{\kappa ,r}{\exp }_{\kappa ,r}^K\left(y\right).\hfill \end{array}$$

(119)

Let us assume that the link function is defined as $f\left(\mathit{w}\right)={\log }_{\kappa ,r}\left(\mathit{w}\right)$ and its inverse (if approximated version is accepted) $f^{(-1)}\left(\mathit{w}\right)={\exp }_{\kappa ,r}\left(\mathit{w}\right)$, then using a general MD Formula (6), and fundamental properties described above, we obtain a general MD formula employing a wide family of deformed logarithms arising from group entropies or trace-form entropies:

$$\boxed{{\displaystyle {\mathit{w}}_{t+1}={\exp }_{\kappa ,r}\left[{\log }_{\kappa ,r}({\mathit{w}}_t)-{\eta }_t\nabla L\left({\mathit{w}}_t\right)\right]={\mathit{w}}_t\hspace{0.33em}{\otimes }_{\kappa ,r}\hspace{0.33em}{\exp }_{\kappa ,r}\left(-{\eta }_t\nabla L(\right.{\mathit{w}}_t))}}$$

(120)

where ${\otimes }_{\kappa ,r}$-multiplication is defined/determined as follows

$$\begin{array}{c}\hfill x\hspace{0.33em}{\otimes }_{\kappa ,r}\hspace{0.33em}y={\exp }_{\kappa ,r}\left({\log }_{\kappa ,r}\left(x\right)\hspace{0.33em}+\hspace{0.33em}{\log }_{\kappa ,r}\left(y\right)\right).\end{array}$$

(121)

Alternatively, due to some complexity of computing precisely ${\exp }_{\kappa ,r}\left(x\right)$, in the general case, we can use formula MMD/NG (10) to derive a quite flexible and general NG gradient update:

$$\boxed{{\displaystyle {\mathit{w}}_{t+1}=[{\mathit{w}}_t-{\eta }_t\mathrm{diag}\left\{{\left(\frac{\mathsf{\partial }{\log }_{\kappa ,r}\left({\mathit{w}}_t\right)}{\mathsf{\partial }{\mathit{w}}_t}\right)}^{-1}\right\}\left(\nabla L(\right.{\mathit{w}}_t\left)\right){]}_{+}}}$$

(122)

where ${\displaystyle \mathrm{diag}\left\{{\left(\frac{\mathsf{\partial }{\log }_{\kappa ,r}\left({\mathit{w}}_t\right)}{\mathsf{\partial }{\mathit{w}}_t}\right)}^{-1}\right\}}$ is a positive-definite diagonal matrix, with the diagonal entries

$$\begin{array}{c}\hfill {\displaystyle {\left(\frac{\mathsf{\partial }{\log }_{\kappa ,r}\left({\mathit{w}}_t\right)}{\mathsf{\partial }{\mathit{w}}_{i,t}}\right)}^{-1}=\frac{2\kappa }{(r+\kappa ){\mathit{w}}_{i,t}^{r+\kappa -1}-(r-\kappa ){\mathit{w}}_{i,t}^{r-\kappa -1}}>0,\hspace{0.33em}-\left|\kappa \right|\le r\le \left|\kappa \right|.}\end{array}$$

(123)

6. Generalization and Normalization of Mirror Descent

Summarizing, all of the GEG updates proposed in this paper can be presented in normalized form (by projecting to unit simplex) in the following general and flexible form

$$\begin{array}{ccc}\hfill {\tilde{\mathit{w}}}_{t+1}& =& {\mathit{w}}_t{\otimes }_D{\exp }_D\left(-{\mathit{\eta }}_t\odot \nabla \widehat{L}\left({\mathit{w}}_t\right)\right)\hspace{0.33em}(\mathrm{Generalized}\hspace{0.33em}\mathrm{multiplicative}\hspace{0.33em}\mathrm{update}),\hfill \end{array}$$

(124)

$$\begin{array}{ccc}\hfill {\mathit{w}}_{t+1}& =& {\displaystyle \frac{{\tilde{\mathit{w}}}_{t+1}}{\left|\right|{\tilde{\mathit{w}}}_{t+1}{\left|\right|}_1}},\hspace{0.33em}(\mathrm{Projection}\hspace{0.33em}\mathrm{to}\hspace{0.33em}\mathrm{unit}\hspace{0.33em}\mathrm{simplex})\hfill \end{array}$$

(125)

where ${\exp }_D\left(x\right)$ (${\log }_D\left(x\right)$) is a generalized exponential (logarithm), $\widehat{L}\left({\mathit{w}}_t\right)=L({\mathit{w}}_t/\left|\right|{\mathit{w}}_t{\left|\right|}_1)$ is normalized/scaled loss function, ${\mathit{\eta }}_t$ is a vector of the learning rates, $\nabla \widehat{L}\left({\mathit{w}}_t\right)=\nabla L\left({\mathit{w}}_t\right)-({\mathit{w}}^T\nabla L\left({\mathit{w}}_t\right))\mathbf{1}$ and the generalized D-multiplication is computed as

$$\begin{array}{c}\hfill {\mathit{w}}_t{\otimes }_D{\exp }_D\left({\mathit{g}}_t\right)={\exp }_D({\log }_D\left({\mathit{w}}_t\right)+{\mathit{g}}_t).\end{array}$$

(126)

Here, ${\log }_D\left({\mathit{w}}_t\right)$ and its inverse ${\exp }_D\left({\mathit{w}}_t\right)$ mean any deformed logarithm and exponential investigated in this paper (i.e., the Tsallis, Kaniadakis, ST, KLS and KS exponential/logarithm).

Alternatively, when the inverse function can not be precisely computed, we can use an MMD/NG additive natural gradient Formula (10), which is expressed in general as

$$\begin{array}{||}\hline \mathrm{(127)}& {\mathit{w}}_{t+1}=[{\mathit{w}}_t-\eta \mathrm{diag}\left\{{\left({\displaystyle \frac{d\hspace{0.33em}{\log }_D\left({\mathit{w}}_t\right)}{d\hspace{0.33em}{\mathit{w}}_t}}\right)}^{-1}\right\}{\nabla }_{\mathit{w}}\widehat{L}\left({\mathit{w}}_t\right){]}_{+},\mathrm{(128)}& {\mathit{w}}_{t+1}={\displaystyle \frac{{\tilde{\mathit{w}}}_{t+1}}{\left|\right|{\tilde{\mathit{w}}}_{t+1}{\left|\right|}_1}},\hspace{0.33em}\hspace{0.33em}{\mathit{w}}_t\in {\mathbb{R}}_{+}^N,\hspace{0.33em}\forall t\\ \hline\end{array}$$

where ${\displaystyle \mathrm{diag}\left\{{\left(\frac{d\hspace{0.33em}{\log }_D\left(\mathit{w}\right)}{d\mathit{w}}\right)}^{-1}\right\}=\mathrm{diag}\left\{{\left(\frac{d\hspace{0.33em}{\log }_D\left(\mathit{w}\right)}{d{\mathit{w}}_1}\right)}^{-1},\dots ,{\left(\frac{d\hspace{0.33em}{\log }_D\left(\mathit{w}\right)}{d{\mathit{w}}_N}\right)}^{-1}\right\}}$ is a diagonal positive-definite matrix.

7. Conclusions and Discussion

This study establishes a comprehensive framework for applying trace-form entropies and associated deformed logarithms in both Mirror Descent and equivalently Generalized Exponentiated Gradient algorithms. By systematically exploring trace-form entropies, especially Tsallis, Kaniadakis, Scarfone, and Sharma–Taneja–Mittal forms as regularization terms, we unveil new families of mirror gradient descent algorithms that can be tailored to the optimization landscape through suitably chosen hyperparameters. The adoption of these generalized entropies opens the door to obtaining advantageous properties such as improved convergence rates, robustness against vanishing/exploding gradients, and inherent flexibility for handling non-Euclidean geometries.

Table 1. Overview of the generalized exponentiated gradient (GEG) updates.

Entropy	Deformed Exponential	MD/GEG Update
Shannon	$\exp \left(x\right)={\sum }_{i=0}^{\infty }{\displaystyle \frac{x^i}{i!}}$	${\mathit{w}}_{t+1}={\mathit{w}}_t\odot \exp (-\eta \nabla L\left({\mathit{w}}_t\right))$ (EG)
Tsallis	${\exp }_q^T\left(x\right)=\left\{\begin{array}{cc}{[1+(1-q)x]}_{+}^{1/(1-q)}& q\ne \hspace{0.33em}1\hfill \\ \\ \exp \left(x\right)& q=1\hfill \end{array}\right.$	${\mathit{w}}_{t+1}={\exp }_q^T\left[{\log }_q^T\left({\mathit{w}}_t\right){\otimes }_q\left(-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right)\right]$
		$={\mathit{w}}_t{\otimes }_q{\exp }_q^T\left(-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right)$ (q-GEG)
Schwämmle-Tsallis	${\exp }_{q,q^{\prime }}^{\mathrm{ST}}\left(x\right)={\left[1+{\displaystyle \frac{1-q}{1-q^{\prime }}}\ln (1+(1-q^{\prime })x)\right]}^{1/(1-q)}$	${\mathit{w}}_{t+1}={\left[{\mathit{w}}_t-\eta \mathrm{diag}\left\{{\mathit{w}}^q\odot \exp \left({\displaystyle \frac{1-q^{\prime }}{1-q}}(1-{\mathit{w}}^{1-q}\right)\right\}{\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right]}_{+}$
Kaniadakis	${\exp }_{\kappa }^K\left(x\right)=\left\{\begin{array}{cc}\mathrm{arcsinh}\left(\kappa x\right)& -1<\kappa <1\hfill \\ \exp \left(x\right)& \kappa =0.\hfill \end{array}\right.$	${\mathit{w}}_{t+1}={\mathit{w}}_t\hspace{0.33em}{\otimes }_{\kappa }\hspace{0.33em}{\exp }_{\kappa }^K\left(-\eta {\nabla }_{\mathit{w}}L\left({\mathit{w}}_t\right)\right)\left(\kappa -\mathrm{GEG}\right)$
KLS	${\exp }_{\kappa ,r}\left(x\right)$	${\mathit{w}}_{t+1}={\left[{\mathit{w}}_t-{\eta }_t\mathrm{diag}\left\{{\left({\displaystyle \frac{\mathsf{\partial }{\log }_{\kappa ,r}\left({\mathit{w}}_t\right)}{\mathsf{\partial }{\mathit{w}}_t}}\right)}^{-1}\right\}\left(\nabla L\left({\mathit{w}}_t\right)\right)\right]}_{+}$
Generic	${\exp }_D\left(x\right)$	${\mathit{w}}_{t+1}={\left[{\mathit{w}}_t-\eta \mathrm{diag}\left\{{\left({\displaystyle \frac{d\hspace{0.33em}{\log }_D\left({\mathit{w}}_t\right)}{d\hspace{0.33em}{\mathit{w}}_t}}\right)}^{-1}\right\}{\nabla }_{\mathit{w}}\widehat{L}\left({\mathit{w}}_t\right)\right]}_{+}$
		${\mathit{w}}_{t+1}={\displaystyle \frac{{\tilde{\mathit{w}}}_{t+1}}{\left|\right|{\tilde{\mathit{w}}}_{t+1}{\left|\right|}_1}}$

summarizes the main results obtained from our study and lists the generalized exponentiated gradient update induced by the deformed exponential functions corresponding to the deformed logarithms used to define the various trace-form entropies.

The theoretical developments presented not only unify additive and multiplicative gradient update rules via Bregman divergences but also pave the way for designing robust machine learning algorithms that have the ability to adapt precisely to the structure of training data distributions via hyperparameters. Future work will investigate broader classes of entropic functions, extending the framework to non-convex and stochastic optimization settings, applying the proposed approach to practical problems, and performing systematic comparisons through computer simulation experiments.