Information-Geometric Models in Data Analysis and Physics

Abstract

Information geometry provides a data-informed geometric lens for understanding data or physical systems, treating data or physical states as points on statistical manifolds endowed with information metrics, such as the Fisher information. Building on this foundation, we develop a robust mathematical framework for analyzing data residing on Riemannian manifolds, integrating geometric insights into information-theoretic principles to reveal how information is structured by curvature and nonlinear manifold geometry. Central to our approach are tools that respect intrinsic geometry: gradient flow lines, exponential and logarithmic maps, and kernel-based principal component analysis. These ingredients enable faithful, low-dimensional representations and insightful visualization of complex data, capturing both local and global relationships that are critical for interpreting physical phenomena, ranging from microscopic to cosmological scales. This framework may elucidate how information manifests in physical systems and how informational principles may constrain or shape dynamical laws. Ultimately, this could lead to groundbreaking discoveries and significant advancements that reshape our understanding of reality itself.

1. Introduction

Information geometry provides insights into understanding data or physical systems by treating data or physical states as points on statistical manifolds, which are equipped with information metrics such as the Fisher information. This viewpoint has become a vibrant interdisciplinary frontier at the crossroads of information and physics, enabling principled analysis of complex data and new information-derived formulations of physical laws [1]. The resulting program promises both conceptual advances—clarifying the link between information and state spaces—and practical benefits for data analysis and the exploration of physical phenomena, ranging from the quantum to the cosmological scale.

Building upon this conceptual foundation, we introduce a robust mathematical framework that leverages advanced geometric and statistical techniques to analyze data residing on Riemannian manifolds [2]. Central to our approach are methods that respect the intrinsic curvature and nonlinear structure of manifold data, such as flow lines of gradient vector fields, exponential and logarithmic maps, and kernel-based principal component analysis (PCA). Throughout the document, we provide a detailed explanation of these tools. They enable accurate, low-dimensional representations and insightful visualization of complex data structures, capturing both the local and global geometric relationships that are critical for understanding the underlying phenomena [3].

By integrating geometric insights with an information-theoretic perspective, our novel framework aims to elucidate the mechanisms by which information emerges in physical systems. This approach reinforces the compelling idea that informational principles are fundamental to the universe’s fabric [4]. It paves the way for exciting possibilities in both pioneering theoretical exploration and significant practical applications across various fields, including data analysis, physics, and more.

The concept that information functions as a fundamental building block of the universe is gaining remarkable momentum among scientists and philosophers [5]. This enlightening perspective suggests that, beyond the realms of matter and energy, information intricately weaves the very fabric of reality, shaping the structure and dynamics of everything from subatomic particles to vast cosmic phenomena. Recognizing information as a core component unlocks deep insights into the nature of physical laws, the origins of the universe, and the intricate interconnectedness of all existence. This paradigm shift not only enhances our understanding but also paves the way for groundbreaking explorations into the universe’s most profound mysteries.

In this initial article, we aim to present a solid mathematical framework and innovative conceptual tools for investigating the deep relationship between information and the physical universe, which has been explored in previous studies [4]. We begin by describing the information sources, the Riemannian structure of the parameter space, and Kernel methods. A significant portion of the article is dedicated to illustrating a data analysis application of the framework, with a particular focus on Kernel PCA. Additionally, we briefly introduce a physical application by expanding upon the variational principle previously published [4]. This information principle will be studied in greater detail in a subsequent manuscript.

By formalizing the principles that govern information processing and its manifestation within physical systems, we aim to uncover the intricate mechanisms through which information may emerge from or be embedded in the fundamental laws of physics. This approach not only establishes strong connections between abstract informational concepts and concrete physical models, but it also provides a solid foundation for investigating how information-theoretic principles could transform our understanding and inspire the development of new theories about the nature of reality.

2. Materials and Methods

In this section, we will meticulously establish a comprehensive mathematical framework by thoroughly defining the information sources and the intricate structure of the parameter space in Section 2.1. We will then examine the rich Riemannian geometry that characterizes this parameter space in Section 2.2, leading to an insightful discussion on the powerful applications of kernel methods in Section 2.3. Section 2.4 examines data analysis in low-dimensional spaces, and Section 2.5 presents a framework for understanding reality.

2.1. The Information Sources and the Parameter Space

Data analysis and physics are inherently connected through their dependence on information sources that supply the data or physical states for examination. However, physical models typically provide a more nuanced and detailed representation of reality, often rooted in causal relationships. With this in mind, we intend to introduce practical tools that can be applied in both fields. Our exploration will begin with an in-depth examination of the vital role of observation, a fundamental aspect that significantly shapes our understanding.

Although the entirety of this phenomenon is located in the observer’s consciousness, it is convenient to simplify the discussion by distinguishing three fundamental elements. The first element is the object studied. The second element is the knowing subject, who is really experimenting: They are aware of what happens. The third element is the environment. These three elements are part of reality in any observation process, and they appear as different elements in the consciousness of the knowing subject, although the reality that underlies and makes observation possible remains hidden. In a simplified sense, we will consider that the said reality is formed by entities that can be viewed as information sources, providing sequences of data for analysis. Some of these entities are used to maintain the underlying structure that makes the internal experience of the knowing subject possible. Let us introduce some notation to establish a convenient and simplified mathematical framework for our work.

To describe the observation process, we will introduce a set, $\mathrm{\Theta }$, which we will call the parameter space. As a first approximation, we can consider it to be an m-dimensional $C^{\infty }$ real manifold, although infinite-dimensional Hilbert or Banach manifolds could also be considered; see [6] for instance. Furthermore, for many purposes, it will suffice to consider the case where $\mathrm{\Theta }$ is a connected open set of ${\mathbb{R}}^m$, and in this case, it is customary to use the same symbol, $\theta$, to denote points and coordinates. Bearing this observation in mind, we will adopt this case and notation from now on to present the results more familiarly, although they could be written in more general terms.

The manifold $\mathrm{\Theta }$ is really a manifold generated by the observer, regardless of whether this is an immediate abstraction from reality is apprehended directly by the observer’s brain–sensory complex or whether it is constructed from it, following well-defined logical procedures. Observe also that the manifold $\mathrm{\Theta }$ has a natural topology induced by its atlas structure, and thus, this topology allows the construction of the Borel $\sigma$-algebra, $\mathcal{B}$, the $\sigma$-algebra induced by the topology, that is, the $\sigma$-algebra generated by the open sets in $\mathrm{\Theta }$. Moreover, in the measurable space $(\mathrm{\Theta },\mathcal{B})$, we may consider a $\sigma$-finite positive measure $\mu$ absolutely continuous with respect to any local Lebesgue measure induced through the atlas structure that we shall take as a reference measure of this manifold.

We shall consider that all relevant aspects of the observation process, determined by the object, the subject, the environment, or a mixture of all of them, can be adequately described by the elements of $\mathrm{\Theta }$ through a family of complex-valued maps that we describe below.

Therefore, to further investigate the behavior and properties of information sources, we will introduce a new tool: a regular parametric family of functions, ${\mathcal{F}}_{\mathrm{\Theta }}$, as a family of measurable maps $h:\mathrm{\Theta }\times \mathrm{\Theta }⟶\mathbb{C}$. These can be expressed in their complex exponential form, that is, $h(\mathit{\omega },\mathit{\theta })=r(\mathit{\omega },\mathit{\theta })e^{i\zeta (\mathit{\omega },\mathit{\theta })}$ for convenient real-valued functions $r(\mathit{\omega },\mathit{\theta })\ge 0$ and $\zeta (\mathit{\omega },\mathit{\theta })$, the modulus and the argument of $h(\mathit{\omega },\mathit{\theta })$, when necessary, and we shall assume that the function $\zeta$ is independent of the reference measure $\mu$; however, if we set $\mathit{\theta }$, the square of the function r, i.e., $r^2(\mathit{\omega },\mathit{\theta })=h(\mathit{\omega },\mathit{\theta })\overline{h(\mathit{\omega },\mathit{\theta })}$ is a probability density with respect to $\mu$, that is, $P_{\mathit{\theta }}\left(d\mathit{\omega }\right)=r^2(\mathit{\omega },\mathit{\theta })\mu \left(d\mathit{\omega }\right)$ is a probability measure in $(\mathrm{\Theta },\mathcal{B})$, $\forall \mathit{\theta }\in \mathrm{\Theta }$. The function $r(·,\mathit{\theta })$ is therefore dependent on the reference measure $\mu$, although the probability measure itself, $P_{\mathit{\theta }}$, is independent. Specifically, if $\nu$ is another reference measure, by the Radon–Nikodym Theorem we shall have that ${\displaystyle \frac{d\mu }{d\nu }}=\mathfrak{q}$ for a convenient measurable positive function $\mathfrak{q}$, then $P_{\mathit{\theta }}\left(d\mathit{\omega }\right)=r^2(\mathit{\omega },\mathit{\theta })\mu \left(d\mathit{\omega }\right)=r^2(\mathit{\omega },\mathit{\theta })\mathfrak{q}\left(\mathit{\omega }\right)\nu \left(d\mathit{\omega }\right)={\widehat{r}}^2(\mathit{\omega },\mathit{\theta })\nu \left(d\mathit{\omega }\right)$, obtaining $\widehat{r}(\mathit{\omega },\mathit{\theta })=r(\mathit{\omega },\mathit{\theta })\sqrt{\mathfrak{q}\left(\mathit{\theta }\right)}$, which explicitly indicates the dependence of r with respect to the reference measure $\mu$. In addition, we will implicitly assume all the regularity conditions that we will require for the development of subsequent calculations, such as the necessary smoothness with respect to $\mathit{\theta }$ and that the closure of the set where the function $r(·,\mathit{\theta })$ is strictly positive is independent of the parameter $\mathit{\theta }$. We refer to $\mu$ as a reference measure of the model, h as the model function, and $f(\mathit{\omega },\mathit{\theta })=r^2(\mathit{\omega },\mathit{\theta })$ as the density of the model below.

2.2. The Riemannian Structure in the Parameter Space

The exploration of Riemannian structures in parameter spaces provides a profound geometric perspective on models and their underlying function spaces. By endowing these spaces with a Riemannian metric, we can investigate the intrinsic geometric properties that influence model behavior, estimation, and inference. Such a perspective enables a more nuanced understanding of the complexity and curvature of the parameter space, offering insights that extend beyond traditional Euclidean approaches.

Letting $L^2(\mathrm{\Theta },\mu )$ be the Hilbert space of these complex-valued measurable functions q in $\mathrm{\Theta }$ such that the Lebesgue integral ${\int }_{\mathrm{\Theta }}{\left|q\left(\mathit{\omega }\right)\right|}^2\mu \left(d\mathit{\omega }\right)<\infty$, where $\left|z\right|=\sqrt{z\overline{z}},z\in \mathbb{C}$, if we denote by $D^2$ the square of the Hilbert distance

$$D^2(p,q)={\int }_{\mathrm{\Theta }}{|p\left(\mathit{\omega }\right)-q\left(\mathit{\omega }\right)|}^2\mu \left(d\mathit{\omega }\right),$$

(1)

once given the model function $h(·,\mathit{\theta })$ of a parametric family, ${\mathcal{F}}_{\mathrm{\Theta }}$, since these functions are naturally identifiable with elements of $L^2(\mathrm{\Theta },\mu )$, we are able to build the function

$$\begin{array}{cc}\hfill {\delta }_{\kappa }^2(\mathit{\gamma },\mathit{\theta })& ={\kappa }^2{D}^2(h(·,\mathit{\gamma }),h(·,\mathit{\theta }))={\kappa }^2{\int }_{\mathrm{\Theta }}{|h(\mathit{\omega },\mathit{\gamma })-h(\mathit{\omega },\mathit{\theta })|}^2\mu \left(d\mathit{\omega }\right)\hfill \\ & ={\kappa }^2\left(2-{\int }_{\mathrm{\Theta }}\left(h(\mathit{\omega },\mathit{\gamma })\overline{h(\mathit{\omega },\mathit{\theta })}+h(\mathit{\omega },\mathit{\theta })\overline{h(\mathit{\omega },\mathit{\gamma })}\right)\mu \left(d\mathit{\omega }\right)\right),\hfill \end{array}$$

(2)

where $\kappa >0$ is a positive constant which determines the units of the dissimilarity measure ${\delta }_{\kappa }^2$ on $\mathrm{\Theta }$; and when $\mathit{\gamma }=\mathit{\theta }$, we have taken into account ${\int }_{\mathrm{\Theta }}h(\mathit{\omega },\mathit{\theta })\overline{h(\mathit{\omega },\mathit{\theta })}\mu \left(d\mathit{\omega }\right)={\int }_{\mathrm{\Theta }}{r}^2(\mathit{\omega },\mathit{\theta })\mu \left(d\mathit{\omega }\right)={\int }_{\mathrm{\Theta }}f(\mathit{\omega },\mathit{\theta })\mu \left(d\mathit{\omega }\right)=1\forall \mathit{\theta }\in \mathrm{\Theta }$. Notice that since $\left|h\right|=|\overline{h}|=r$, if we change the reference measure $\mu$ to $\nu$ accordingly, r changes to $\widehat{r}$, and when we integrate (2) now with respect to $\nu$, we find that ${\delta }^2(\mathit{\gamma },\mathit{\theta })$ remains invariant.

We are going to consider the Riemannian metric induced in $\mathrm{\Theta }$ by ${\delta }_{\kappa }^2$ following the same basic ideas of [7]. If we define $d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\gamma }\right)={\delta }_{\kappa }^2(\mathit{\gamma },\mathit{\theta })$, we will have $d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\theta }\right)=0$, and, taking into account that the Jacobian of $d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\gamma }\right)$ in $\mathit{\gamma }=\mathit{\theta }$ is also null, since the function has an absolute minimum at $\mathit{\theta }$, the second-order expansion of $d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\gamma }\right)$ at $\mathit{\theta }$ will be given by its Hessian at $\mathit{\theta }$, ${\left(d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\theta }\right)\right)}^{″}={\left({\left.{\displaystyle \frac{{\partial }^2{d}_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^i\partial {\gamma }^j}}\right|}_{\mathit{\gamma }=\mathit{\theta }}\right)}_{(m\times m)}$, which will be positive definite or semidefinite. Using matrix notation, we will have

$$d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\gamma }\right)={\displaystyle \frac{1}{2}}{(\mathit{\gamma }-\mathit{\theta })}^{\mathrm{T}}{\left(d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\theta }\right)\right)}^{″}{(\mathit{\gamma }-\mathit{\theta })+O\left(\right|\mathit{\gamma }-\mathit{\theta }|}^3).$$

(3)

Notice also that if we consider an appropriate smooth coordinate change given by $\mathit{\gamma }=({\gamma }^1,\dots ,{\gamma }^m)\mapsto \overline{\mathit{\gamma }}=({\overline{\gamma }}^1,\dots ,{\overline{\gamma }}^m)$, the function $d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\gamma }\right)$ expressed under the new coordinate system as ${\overline{d}}_{\kappa ,\overline{\mathit{\theta }}}^2\left(\overline{\mathit{\gamma }}\right)$ is an invariant, that is, $d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\gamma }\right)={\overline{d}}_{\kappa ,\overline{\mathit{\theta }}}^2\left(\overline{\mathit{\gamma }}\right)$, and we will have, in classical notation, without using the repeated index summation convention criteria unless we explicitly state otherwise, by the chain rule,

$${\displaystyle \frac{\partial {\overline{d}}_{\kappa ,\overline{\mathit{\theta }}}^2}{\partial {\overline{\gamma }}^i}}=\sum _{\alpha =1}^m{\displaystyle \frac{\partial d_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\alpha }}}{\displaystyle \frac{\partial {\gamma }^{\alpha }}{\partial {\overline{\gamma }}^i}}.$$

(4)

Moreover, since the second-order partial derivatives are

$${\displaystyle \frac{{\partial }^2{\overline{d}}_{\kappa ,\overline{\mathit{\theta }}}^2}{\partial {\overline{\gamma }}^j\partial {\overline{\gamma }}^i}}=\sum _{\alpha =1}^m\sum _{\beta =1}^m{\displaystyle \frac{{\partial }^2{d}_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}{\displaystyle \frac{\partial {\gamma }^{\beta }}{\partial {\overline{\gamma }}^j}}{\displaystyle \frac{\partial {\gamma }^{\alpha }}{\partial {\overline{\gamma }}^i}}+\sum _{\alpha =1}^m{\displaystyle \frac{\partial d_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\alpha }}}{\displaystyle \frac{{\partial }^2{\gamma }^{\alpha }}{\partial {\overline{\gamma }}^j\partial {\overline{\gamma }}^i}},$$

(5)

when evaluating them at $\mathit{\gamma }=\mathit{\theta }$, we will have $\partial d_{\kappa ,\mathit{\theta }}^2/\partial {\gamma }^{\alpha }{|}_{\mathit{\gamma }=\mathit{\theta }}=0$. Therefore,

$${\left.{\displaystyle \frac{{\partial }^2{\overline{d}}_{\kappa ,\overline{\mathit{\theta }}}^2}{\partial {\overline{\gamma }}^j\partial {\overline{\gamma }}^i}}\right|}_{\overline{\mathit{\gamma }}=\overline{\mathit{\theta }}}=\sum _{\alpha =1}^m\sum _{\beta =1}^m{\left.{\displaystyle \frac{{\partial }^2{d}_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}\right|}_{\mathit{\gamma }=\mathit{\theta }\left(\overline{\mathit{\theta }}\right)}{\left.{\displaystyle \frac{\partial {\gamma }^{\beta }}{\partial {\overline{\gamma }}^j}}\right|}_{\overline{\mathit{\gamma }}=\overline{\mathit{\theta }}}{\left.{\displaystyle \frac{\partial {\gamma }^{\alpha }}{\partial {\overline{\gamma }}^i}}\right|}_{\overline{\mathit{\gamma }}=\overline{\mathit{\theta }}},$$

(6)

where we have denoted by $\mathit{\theta }\left(\overline{\mathit{\theta }}\right)$ the coordinates of the point $\mathit{\theta }$ expressed under the transformed coordinate system, obtaining that the components of the Hessian ${\left(d_{\kappa ,\mathit{\theta }}^2\left(\mathit{\gamma }\right)\right)}^{″}$ at $\mathit{\gamma }=\mathit{\theta }$ are the components of a second-order covariant symmetric tensor in the tangent space $T_{\mathit{\theta }}\left(\mathrm{\Theta }\right)$; see [8,9] and Appendix A for further details. This tensor is at least semidefinite positive, but hereafter, we will assume the case that is positive definite, as in most of the interesting examples that we will consider. Then, it may be used to define a scalar product at each tangent space, and thus a Riemannian metric in the whole manifold $\mathrm{\Theta }$. The metric tensor components, under the coordinates $\mathit{\theta }=({\theta }^1,\dots ,{\theta }^m)$, will be given by

$$g_{ij}\left(\mathit{\theta }\right)={\displaystyle \frac{1}{2}}{\displaystyle \frac{{\partial }^2{d}_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^i\partial {\gamma }^j}}{|}_{\mathit{\gamma }=\mathit{\theta }}i,j=1,\dots ,m,$$

(7)

i.e., the components of the metric tensor will be equal to one half times the second partial derivatives of the function $d_{\kappa ,\mathit{\theta }}^2$, and the line element corresponding to the above-mentioned induced Riemannian metric, under $\mathit{\theta }=({\theta }^1,\dots ,{\theta }^m)$, is given through its square by

$$d{s}^2=\sum _{i=1}^m\sum _{j=1}^m{g}_{ij}\left(\mathit{\theta }\right)d{\theta }^{i}d{\theta }^j.$$

(8)

This formulation is typical in classical information geometry, where the metric is derived from second derivatives of divergence functions or similar measures.

If we express the complex-valued function $h(·,\mathit{\theta })$ in its exponential form, that is, $h(·,\mathit{\theta })=r(·,,\mathit{\theta })e^{i\zeta (·,\mathit{\theta })}$ for a convenient real-valued function $\zeta$ of $(·,\mathit{\theta })$, from (2), taking into account that if $z=r{e}^{i\zeta }$ then $\overline{z}=r{e}^{-i\zeta }$, we obtain

$$\begin{array}{cc}\hfill {\delta }_{\kappa }^2(\mathit{\gamma },\mathit{\theta })& ={\kappa }^2\left(2-{\int }_{\mathrm{\Theta }}(r(\omega ,\mathit{\gamma })e^{i\zeta (\omega ,\mathit{\gamma })}r(\omega ,\mathit{\theta })e^{-i\zeta (\omega ,\mathit{\theta })}\right.\hfill \\ & \left.+r(\omega ,\mathit{\theta })e^{i\zeta (\omega ,\mathit{\theta })}r(\omega ,\mathit{\gamma })e^{-i\zeta (\omega ,\mathit{\gamma })})\mu \left(d\mathit{\omega }\right)\right)\hfill \\ \hfill & ={\kappa }^2\left(2-{\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })\left(e^{i\left(\zeta \right(\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta }\left)\right)}+e^{-i\left(\zeta \right(\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta }\left)\right)}\right)\mu \left(d\mathit{\omega }\right)\right)\hfill \\ \hfill & =2{\kappa }^2\left(1-{\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })cos\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta })\right)\mu \left(d\mathit{\omega }\right)\right),\hfill \end{array}$$

(9)

and then, if we denote by $d_H^2$ the square of the normalized Hellinger distance between the densities $f(\omega ,\mathit{\gamma })$ and $f(\omega ,\mathit{\theta })$

$$\begin{array}{cc}\hfill d_H^2(\mathit{\gamma },\mathit{\theta })& ={\displaystyle \frac{1}{2}}{\int }_{\mathrm{\Theta }}{\left(\sqrt{f(\omega ,\mathit{\gamma })}-\sqrt{f(\omega ,\mathit{\theta })}\right)}^2\mu \left(d\mathit{\omega }\right)\hfill \\ & ={\displaystyle \frac{1}{2}}{\int }_{\mathrm{\Theta }}{\left(r(\omega ,\mathit{\gamma })-r(\omega ,\mathit{\theta })\right)}^2\mu \left(d\mathit{\omega }\right)\hfill \\ & =1-{\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })\mu \left(d\mathit{\omega }\right),\hfill \end{array}$$

(10)

since $d_H^2(\mathit{\gamma },\mathit{\theta })\ge 0$, we have

$$0\le {\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })\mu \left(d\mathit{\omega }\right)=1-d_H^2(\mathit{\gamma },\mathit{\theta }).$$

(11)

Taking into account that ${\delta }_{\kappa }^2(\mathit{\gamma },\mathit{\theta })\ge 0$ and $-1\le cos\left(u\right)\le 1$, we shall have

$$\begin{array}{cc}\hfill \left|{\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })cos(\zeta (\omega ,\mathit{\gamma })\right.\left.-\zeta (\omega ,\mathit{\theta })\left)\mu \right(d\mathit{\omega })\right|& \le {\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })\mu \left(d\mathit{\omega }\right)\hfill \\ & =1-d_H^2(\mathit{\gamma },\mathit{\theta }).\hfill \end{array}$$

(12)

Therefore, from (12) we have

$$\begin{array}{cc}\hfill d_H^2(\mathit{\gamma },\mathit{\theta })& \le 1-{\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })cos\left(r(\omega ,\mathit{\gamma })-r(\omega ,\mathit{\theta })\right)\mu \left(d\mathit{\gamma }\right)\hfill \\ & \le 2-d_H^2(\mathit{\gamma },\mathit{\theta }),\hfill \end{array}$$

(13)

and we finally obtain

$$2{\kappa }^2{d}_H^2(\mathit{\gamma },\mathit{\theta })\le {\delta }_{\kappa }^2(\mathit{\gamma },\mathit{\theta })\le 2{\kappa }^2\left(2-d_H^2(\mathit{\gamma },\mathit{\theta })\right).$$

(14)

Observe, additionally, that for $\kappa =2$, the quantity $8d_H^2(\mathit{\gamma },\mathit{\theta })$ is the square of the distance between $2\sqrt{f(·,\mathit{\gamma })}$ and $2\sqrt{f(·,\mathit{\theta })}$, functions that are on a radius 2 sphere of $L^2(\mathrm{\Theta },\mu )$. Additionally, the aforementioned amount is, locally, the information metric on $\mathrm{\Theta }$; for instance, see [7].

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial d_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\alpha }}}& =-2{\kappa }^2\left({\int }_{\mathrm{\Theta }}\right({\displaystyle \frac{\partial r}{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })r(\mathit{\omega },\mathit{\theta })cos\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta })\right)\hfill \\ & -r(\mathit{\omega },\mathit{\gamma })r(\mathit{\omega },\mathit{\theta })sin\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta }){\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right)\mu \left(d\mathit{\omega }\right)\left)\right).\hfill \end{array}$$

(15)

At the point $\mathit{\gamma }=\mathit{\theta }$, we have

$$\begin{array}{c}\hfill {\left.{\displaystyle \frac{\partial d_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\alpha }}}\right|}_{\mathit{\gamma }=\mathit{\theta }}=-2{\kappa }^2\left({\int }_{\mathrm{\Theta }}{\left.{\displaystyle \frac{\partial r}{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}r(\mathit{\omega },\mathit{\theta })\mu \left(d\mathit{\omega }\right)\right)=0,\end{array}$$

(16)

since ${\int }_{\mathrm{\Theta }}{r}^2(\mathit{\omega },\mathit{\theta })\mu \left(d\mathit{\omega }\right)=1$, and any partial derivative with respect to ${\theta }^{\alpha }$ is equal to zero.

Additionally, since the second-order partial derivatives of $d_{\kappa ,\mathit{\theta }}^2$ divided by 2 are

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{2}}{\displaystyle \frac{{\partial }^2{d}_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}=& -{\kappa }^2\left({\int }_{\mathrm{\Theta }}\right({\displaystyle \frac{{\partial }^{2}r}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })r(\mathit{\omega },\mathit{\theta })cos\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta })\right)\hfill \\ \hfill & -{\displaystyle \frac{\partial r}{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })r(\mathit{\omega },\mathit{\theta })sin\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta })\right){\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma })\hfill \\ \hfill & -{\displaystyle \frac{\partial r}{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma })r(\mathit{\omega },\mathit{\theta })sin\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta })\right){\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\hfill \\ \hfill & -r(\mathit{\omega },\mathit{\gamma })r(\mathit{\omega },\mathit{\theta })cos\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta }){\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma }){\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma })\right)\hfill \\ \hfill & -r(\mathit{\omega },\mathit{\gamma })r(\mathit{\omega },\mathit{\theta })sin\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta }){\displaystyle \frac{{\partial }^2\zeta }{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right)\mu \left(d\mathit{\omega }\right)\left)\right),\hfill \end{array}$$

(17)

at the point $\mathit{\gamma }=\mathit{\theta }$, we have

$$\begin{array}{cc}\hfill {\left.{\displaystyle \frac{1}{2}}{\displaystyle \frac{{\partial }^2{d}_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}\right|}_{\mathit{\gamma }=\mathit{\theta }}=& -{\kappa }^2\left({\int }_{\mathrm{\Theta }}\right({\left.{\displaystyle \frac{{\partial }^{2}r}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}r(\mathit{\omega },\mathit{\theta })\hfill \\ \hfill & -r^2(\mathit{\omega },\mathit{\theta }){\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}{\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma }))\right|}_{\mathit{\gamma }=\mathit{\theta }}\left)\mu \left(d\mathit{\omega }\right)\right).\hfill \end{array}$$

(18)

However, if we take into account that $r(\mathit{\omega },\mathit{\theta })=\sqrt{f(\mathit{\omega },\mathit{\theta })}$ is the real square root of a probability density with respect to the reference measure $\mu$, we shall have

$$\begin{array}{cc}\hfill {\left.{\displaystyle \frac{1}{2}}{\displaystyle \frac{{\partial }^2{d}_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}\right|}_{\mathit{\gamma }=\mathit{\theta }}=& -{\kappa }^2\left({\int }_{\mathrm{\Theta }}\right({\left.{\displaystyle \frac{{\partial }^2\sqrt{f}}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}\sqrt{f(\mathit{\omega },\mathit{\theta })}\hfill \\ \hfill & -f(\mathit{\omega },\mathit{\theta }){\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}{\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma }))\right|}_{\mathit{\gamma }=\mathit{\theta }}\left)\mu \left(d\mathit{\omega }\right)\right),\hfill \end{array}$$

(19)

and since

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial \sqrt{f}}{\partial {\theta }^{\alpha }}}={\displaystyle \frac{1}{2}}{\displaystyle \frac{\partial lnf}{\partial {\theta }^{\alpha }}}\sqrt{f}\mathrm{and}{\displaystyle \frac{{\partial }^2\sqrt{f}}{\partial {\theta }^{\beta }\partial {\theta }^{\alpha }}}=-{\displaystyle \frac{1}{4}}{\displaystyle \frac{\partial lnf}{\partial {\theta }^{\beta }}}{\displaystyle \frac{\partial lnf}{\partial {\theta }^{\alpha }}}\sqrt{f}+{\displaystyle \frac{1}{2\sqrt{f}}}{\displaystyle \frac{{\partial }^{2}f}{\partial {\theta }^{\beta }\partial {\theta }^{\alpha }}},\end{array}$$

(20)

we have

$$\begin{array}{cc}\hfill {\left.{\displaystyle \frac{1}{2}}{\displaystyle \frac{{\partial }^2{d}_{\kappa ,\mathit{\theta }}^2}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}\right|}_{\mathit{\gamma }=\mathit{\theta }}=& {\displaystyle \frac{{\kappa }^2}{4}}{\int }_{\mathrm{\Theta }}({\left.{\displaystyle \frac{\partial lnf}{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}{\left.{\displaystyle \frac{\partial lnf}{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}f(\mathit{\omega },\mathit{\gamma })\hfill \\ \hfill -& {\left.2{\displaystyle \frac{{\partial }^{2}f}{\partial {\gamma }^{\beta }\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}+f(\mathit{\omega },\mathit{\gamma }){\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}{\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }})\mu \left(d\mathit{\omega }\right)\hfill \\ \hfill =& {\displaystyle \frac{{\kappa }^2}{4}}({\int }_{\mathrm{\Theta }}{\left.{\displaystyle \frac{\partial lnf}{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}{\left.{\displaystyle \frac{\partial lnf}{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\theta })\right|}_{\mathit{\gamma }=\mathit{\theta }}f(\mathit{\omega },\mathit{\gamma })\mu \left(d\mathit{\omega }\right)\hfill \\ \hfill +& 4{\int }_{\mathrm{\Theta }}{\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}{\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}f(\mathit{\omega },\mathit{\gamma })\mu \left(d\mathit{\omega }\right)).\hfill \end{array}$$

(21)

Moreover, let $\mathcal{I}=\left(i_{\alpha \beta }\right)$ be the $(m\times m)$ Fisher information matrix corresponding to the density of the model, that is, $i_{\alpha \beta }={\int }_{\mathrm{\Theta }}{\displaystyle \frac{\partial lnf}{\partial {\theta }^{\alpha }}}{\displaystyle \frac{\partial lnf}{\partial {\theta }^{\beta }}}fd\mu$, and additionally define

$${\varkappa }_{\alpha \beta }\left(\mathit{\theta }\right)=4{\int }_{\mathrm{\Theta }}{\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\alpha }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}{\left.{\displaystyle \frac{\partial \zeta }{\partial {\gamma }^{\beta }}}(\mathit{\omega },\mathit{\gamma })\right|}_{\mathit{\gamma }=\mathit{\theta }}f(\mathit{\omega },\mathit{\gamma })\mu \left(d\mathit{\omega }\right).$$

(22)

We obtain that the fundamental tensor of $\mathrm{\Theta }$, (7), converting it as a Riemannian manifold, is equal to

$$g_{\alpha \beta }\left(\mathit{\theta }\right)={\displaystyle \frac{{\kappa }^2}{4}}\left(i_{\alpha \beta }\left(\mathit{\theta }\right)+{\varkappa }_{\alpha \beta }\left(\mathit{\theta }\right)\right).$$

(23)

The Riemannian metric, using repeated index summation convention, can also be expressed as

$$d{s}^2=g_{\alpha \beta }d{\theta }^{\alpha }d{\theta }^{\beta }={\displaystyle \frac{{\kappa }^2}{4}}\left(i_{\alpha \beta }d{\theta }^{\alpha }d{\theta }^{\beta }+{\varkappa }_{\alpha \beta }d{\theta }^{\alpha }d{\theta }^{\beta }\right)={\displaystyle \frac{{\kappa }^2}{4}}\left(d{s}_I^2+d{s}_A^2\right),$$

(24)

where $d{s}_I^2$ is the information metric, and $d{s}_A^2$ is a metric specifically related to the imaginary part of the function that defines the model. Observe that locally the Riemannian metric induced in $\mathrm{\Theta }$ is the distance induced by the Hilbert metric structure in $L^2(\mathrm{\Theta },d\mu )$ on a radius 2 sphere and that it coincides, when $\kappa =2$, with the information metric when $\zeta$ is constant. More results on the information metric can be found in [7,10,11], among others. Additional aspects of the information metric are briefly recalled in Appendix A.

This structure suggests a Kähler-like or complex Riemannian geometry, often encountered in quantum information geometry, where complex structures naturally appear, and the metric encodes both probabilistic and phase information. Quantum information geometry explores the geometric aspects of quantum states and their transformations. Unlike classical probability spaces, quantum state spaces are inherently complex and require a framework that can incorporate both the probabilistic nature of quantum mechanics and the phase information, which is crucial for phenomena like interference and entanglement. The complex structure naturally appears because quantum states are represented by vectors in complex Hilbert spaces, and their evolution and measurement outcomes depend on both amplitude and phase.

The metric in this geometric setting is not just a measure of “distance” between states; it encodes multiple layers of information. On one hand, it captures probabilistic information—how distinguishable two quantum states are based on measurement outcomes—mirroring classical statistical distances but adapted to quantum mechanics. On the other hand, it also encodes phase information, reflecting the relative quantum phases that influence interference effects and the dynamics of quantum systems.

By combining these elements, the described structure offers a comprehensive geometric language that unifies the probabilistic and phase aspects of quantum states. This unified approach facilitates an understanding of quantum state transformations, optimal measurements, and quantum information processing tasks. The resemblance to Kähler geometry provides powerful mathematical tools, such as complex differential calculus and symplectic geometry, which facilitate the analysis of the complex landscape of quantum states and their evolutions. Overall, this geometric perspective enhances our conceptual and computational grasp of quantum phenomena, revealing deep connections between geometry, probability, and phase in the quantum realm.

Please note that while we assume that all properties of an information source are determined by $\mathit{\theta }$ or $h(·,\mathit{\theta })$, the exact value of $\mathit{\theta }$ has to be estimated from the data generated by the source. Although we assume that these data can remain partially hidden, it should still allow a reasonably good estimate of both $\mathit{\theta }$ and $h(·,\mathit{\theta })$.

2.3. Kernel Methods

Kernel methods have become a cornerstone of modern machine learning, enabling flexible and powerful approaches to non-linear data analysis. By implicitly mapping data into high-dimensional (often infinite-dimensional) feature spaces, kernel methods enable linear algorithms to capture intricate patterns that would be difficult to model directly in the original input space. This approach not only enhances the expressive power of models, but it also provides computational advantages through the so-called “kernel trick,” which allows inner products in feature space to be computed efficiently without explicit transformation. At the heart of kernel methods lies the concept of defining kernels based on underlying parametric families.

Given a regular parametric family ${\mathcal{F}}_{\mathrm{\Theta }}$, we can define a natural kernel in $\mathrm{\Theta }$, using $L^2(\mathrm{\Theta },\mu )$ as a feature space, the Hilbert space of measurable functions with complex value and square integrability in $\mathrm{\Theta }$, and $\mathcal{J}$ as a feature map, which maps $\mathrm{\Theta }$ to $L^2(\mathrm{\Theta },\mu )$, defined as $\mathcal{J}\left(\mathit{\theta }\right)=h(·,\mathit{\theta })$. Then, we can define a kernel in $\mathrm{\Theta }$ as

$$K(\mathit{\theta },\mathit{\gamma })={\kappa }^2{〈\mathcal{J}\left(\mathit{\theta }\right),\mathcal{J}\left(\mathit{\gamma }\right)〉}_{L^2(\mathrm{\Theta },\mu )}={\kappa }^2{\int }_{\mathrm{\Theta }}h(\mathit{\omega },\mathit{\gamma })\overline{h(\mathit{\omega },\mathit{\theta })}\mu \left(d\mathit{\omega }\right),$$

(25)

for a convenient positive proportionality constant $\kappa >0$.

A kernel can be interpreted as a measure of similarity between points in the parameter space, and it allows us to identify, though a kernel K, each $\mathit{\theta }\in \mathrm{\Theta }$ with a complex-valued function, $K(·,\mathit{\theta })$, which we assume to contain all relevant features of the point $\mathit{\theta }$, or, more formally, through a function $\mathit{\varphi }$, which we refer to as the canonical feature map, defined as

$$\begin{array}{ccc}\hfill \mathit{\varphi }:\mathrm{\Theta }& \to & {\mathbb{C}}^{\mathrm{\Theta }}=\left\{f\right|f:\mathrm{\Theta }\to \mathbb{C}\}\hfill \\ \hfill \mathit{\theta }& \mapsto & \mathit{\varphi }\left(\mathit{\theta }\right)=K(·,\mathit{\theta }).\hfill \end{array}$$

(26)

Observe that $\mathit{\varphi }\left(\mathit{\theta }\right)$ is properly a function, much simpler than the previously mentioned $\mathcal{J}\left(\mathit{\theta }\right)$, which is an element of $L^2(\mathrm{\Theta },\mu )$ and, therefore, an equivalent class of functions, although it can be identified with one of the equivalence classes to which it belongs. We will consider hereafter that the parametric family ${\mathcal{F}}_{\mathrm{\Theta }}$ is such that the corresponding map $\mathit{\varphi }$, with the natural kernel (25), is 1–1, i.e., $K(·,\mathit{\theta })=K(·,\mathit{\gamma })$ then $\mathit{\theta }=\mathit{\gamma }$, as happens in most cases. Then, we can identify $\mathrm{\Theta }$ with $\mathit{\varphi }\left(\mathrm{\Theta }\right)$ for almost all pursuit purposes. Notice that $\mathit{\varphi }\left(\mathrm{\Theta }\right)$, which we will call the feature manifold, inherits a natural m-dimensional differentiable manifold structure induced by the manifold structures of $\mathrm{\Theta }$. The parameter $\mathit{\theta }$, identifiable with $\mathit{\varphi }\left(\mathit{\theta }\right)$, characterizes what we shall call the state of the observation process and can be expressed using the complex exponential form when necessary.

At this point, it is convenient to outline the following well-known mathematical construction that will lead us to a simpler feature space. See [12,13] for details. We start by recalling the concept of a positive definite kernel. A map $K:\mathrm{\Theta }\times \mathrm{\Theta }\to \mathbb{C}$ is a positive definite kernel if and only if, for any mutually distinct points ${\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_n\in \mathrm{\Theta }$ and for any scalars $z_1,\dots ,z_n\in \mathbb{C}$, where n is an arbitrary positive integer, we have that ${\sum }_{i,j=1}^n{z}_i\overline{z_j}K({\mathit{\theta }}_i,{\mathit{\theta }}_j)\ge 0$where $\overline{z_j}$ is the complex conjugate of $z_j$. Notice that the Hermitian property, $K(\mathit{\theta },\mathit{\gamma })=\overline{K(\mathit{\gamma },\mathit{\theta })}$, is a consequence of the positive-definiteness. Moreover, if for mutually distinct ${\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_n$, the equality only holds for $z_1,=\dots =z_n=0$, it is often said that K is a strictly positive definite Hermitian kernel, although there are small variations in that name to designate this property according different authors. It is also possible to consider real-valued kernels satisfying $K:\mathrm{\Theta }\times \mathrm{\Theta }\to \mathbb{R}$ such that $K(\mathit{\theta },\mathit{\gamma })=K(\mathit{\gamma },\mathit{\theta })$ and ${\sum }_{i,j=1}^n{\alpha }_i{\alpha }_{j}K({\mathit{\theta }}_i,{\mathit{\theta }}_j)\ge 0$ for any positive integer $n\in \mathbb{N}$, ${\alpha }_i\in \mathbb{R}$, ${\mathit{\theta }}_i\in \mathrm{\Theta }$$i=1,\dots ,n$. Observe that (25) satisfies all of the above-mentioned properties, and is thus, in addition, a positive definite kernel because $K(\mathit{\theta },\mathit{\gamma })=\overline{K(\mathit{\gamma },\mathit{\theta })}$, since clearly $\overline{{\int }_{\mathrm{\Theta }}q\left(\mathit{\omega }\right)\mu \left(d\mathit{\omega }\right)}={\int }_{\mathrm{\Theta }}\overline{q\left(\mathit{\omega }\right)}\mu \left(d\mathit{\omega }\right)$ and for any scalars $z_1,\dots ,z_n\in \mathbb{C}$, where n is an arbitrary positive integer, ${\sum }_{i,j=1}^n{z}_i\overline{z_j}K({\mathit{\theta }}_{\mathbf{i}},{\mathit{\theta }}_{\mathbf{j}})={\kappa }^2{\sum }_{i,j=1}^n{z}_i\overline{z_j}{\int }_{\mathrm{\Theta }}h(\mathit{\omega },{\mathit{\theta }}_i)\overline{h(\mathit{\omega },{\mathit{\theta }}_j)}\mu \left(d\mathit{\omega }\right)={\kappa }^2{\int }_{\mathrm{\Theta }}\left({\sum }_{i=1}^n{z}_{i}h(\mathit{\omega },{\mathit{\theta }}_i)\right)\overline{\left({\sum }_{j=1}^n{z}_{j}h(\mathit{\omega },{\mathit{\theta }}_j)\right)}\mu \left(d\mathit{\omega }\right)\ge 0$. Therefore, $K(\mathit{\theta },\mathit{\gamma })$ it is a complex-valued Hermitian kernel.

Therefore, taking into account that $\mathit{\varphi }\left(\mathrm{\Theta }\right)\subset {\mathbb{C}}^{\mathrm{\Theta }}$ and that ${\mathbb{C}}^{\mathrm{\Theta }}$ has a natural vector space structure over $\mathbb{C}$, corresponding to each kernel K, we can build a vector space ${\mathcal{P}}_K=span\left(\mathit{\varphi }\left(\mathrm{\Theta }\right)\right)\subset {\mathbb{C}}^{\mathrm{\Theta }}$, i.e., the set of all finite linear combinations of elements of $\mathit{\varphi }\left(\mathrm{\Theta }\right)$. Observe also that the elements of ${\mathcal{P}}_K$ are complex-valued functions f with domains the parameter space, $\mathrm{\Theta }$, which can be expressed as $f(·)={\sum }_{i=1}^n{z}_{i}K(·,{\mathit{\theta }}_i)$ for a convenient $n\in \mathbb{N}$ and ${\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_n\in \mathrm{\Theta }$, $z_1,\dots ,z_n\in \mathbb{C}$.

Moreover, the kernel K allows one to define a Hermitian inner product $\langle ·,·\rangle$ in ${\mathcal{P}}_K$ as follows: If $f,g\in {\mathcal{P}}_K$, then $f(·)={\sum }_{i=1}^{n_1}{\eta }_{i}K(·,{\mathit{\theta }}_i)$ and $g(·)={\sum }_{j=1}^{n_2}{\varsigma }_{j}K(·,{\mathit{\gamma }}_j)$, with the Hermitian inner product defined as

$$\langle f,g\rangle =\sum _{i=1}^{n_1}\sum _{j=1}^{n_2}{\eta }_i{\varsigma }_{j}K({\mathit{\gamma }}_j,{\mathit{\theta }}_i),$$

(27)

which turns ${\mathcal{P}}_K$ into a pre-Hilbert space. After a standard completion, we can turn our pre-Hilbert vector space ${\mathcal{P}}_K$ into a true Hilbert space ${\mathcal{H}}_K$, which is also a norm and a metric space with the standard norm and the distance defined as from (27). Moreover, this Hilbert space is the reproducing kernel Hilbert space (RKHS) induced by the kernel function K, see [13,14], which satisfies the so-called reproducing property, i.e., $\langle f,K(·,\mathit{\theta })\rangle =f\left(\mathit{\theta }\right)$. Therefore, $\langle \mathit{\varphi }\left(\mathit{\theta }\right),\mathit{\varphi }\left(\mathit{\gamma }\right)\rangle =\langle K(·,\mathit{\theta }),K(·,\mathit{\gamma })\rangle =K(\mathit{\gamma },\mathit{\theta })$. Additionally, it is a functional Hilbert space, i.e., a Hilbert space in which elements are functions (on $\mathrm{\Theta }$) such that the point evaluation functional ${\mathcal{E}}_{\mathit{\theta }}\left(f\right)=\langle f,K(·,\mathit{\theta })\rangle =f\left(\mathit{\theta }\right)$ is a continuous linear functional on ${\mathcal{H}}_K$, with norm $|{\mathcal{E}}_{\mathit{\theta }}|=\sqrt{K(\mathit{\theta },\mathit{\theta })}=\kappa$, because for the natural kernel (25) we have $K(\mathit{\theta },\mathit{\theta })={\kappa }^2$ since ${\int }_{\mathrm{\Theta }}{\left|h(\mathit{\omega },\mathit{\theta })\right|}^2\mu \left(d\mathit{\omega }\right)=1$. Let us recall now that with additional assumptions about the parameter space from measure and topological theory, many other properties of kernels can be established, like Mercer’s theorem and relationships of square-integrable function spaces to RKHS, via linear operators; see, for instance, ref. [13]. Notice also that the distance in the feature space, $d_{{\mathcal{H}}_K}$, induces a distance in the parameter space, $d_{\mathrm{\Theta }}$, through the map $\mathit{\varphi }$ such that

$$\begin{array}{cc}\hfill d_{\mathrm{\Theta }}^2(\mathit{\theta },\mathit{\gamma })& =d_{{\mathcal{H}}_K}^2(\mathit{\varphi }\left(\mathit{\theta }\right),\mathit{\varphi }\left(\mathit{\gamma }\right))=2{\kappa }^2-K(\mathit{\theta },\mathit{\gamma })-K(\mathit{\gamma },\mathit{\theta })\hfill \\ \hfill & =2{\kappa }^2-{\kappa }^2{\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })e^{i\left(\zeta \right(\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta }\left)\right)}\mu \left(d\mathit{\omega }\right))\hfill \\ & -{\kappa }^2{\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\theta })r(\omega ,\mathit{\gamma })e^{-i\left(\zeta \right(\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta }\left)\right)}\mu \left(d\mathit{\omega }\right))\hfill \\ \hfill & =2{\kappa }^2\left(1-{\int }_{\mathrm{\Theta }}r(\omega ,\mathit{\gamma })r(\omega ,\mathit{\theta })cos\left(\zeta (\omega ,\mathit{\gamma })-\zeta (\omega ,\mathit{\theta })\right)\mu \left(d\mathit{\omega }\right)\right),\hfill \end{array}$$

(28)

which is equal to (9). Observe that this distance reflects the differences between the parameter space points—$\mathit{\theta },\mathit{\gamma }\in \mathrm{\Theta }$—or, equivalently, between their images—$\mathit{\varphi }\left(\mathit{\theta }\right),\mathit{\varphi }\left(\mathit{\gamma }\right)\in \mathit{\varphi }\left(\mathrm{\Theta }\right)$—and locally defines in any of these manifolds $\mathrm{\Theta }$ or $\mathit{\varphi }\left(\mathrm{\Theta }\right)$, the Riemannian metric (24), when $\kappa =2$.

We finally remark that the obtained RKHS, ${\mathcal{H}}_K$, which can be identified as a closed subspace of $L^2(\mathrm{\Theta },\mu )$, will also be considered as a feature space of the kernel K, defined in (25), and can be considered as the smallest feature space corresponding to this kernel. Indeed, if $\mathcal{W}$ is another Hilbert space that allows us to obtain the same kernel K, as in $\mathcal{W}=L^2(\mathrm{\Theta },\mu )$, then there exists a canonical metric surjection from $\mathcal{W}$ onto ${\mathcal{H}}_K$. As an aside, in later physical applications, we will think of this RKHS as the most convenient abstract model for identifying objects in the physical world, viewing them as vectors in this infinite-dimensional Hilbert space.

Therefore, at this moment, a point in the parameter space $\mathrm{\Theta }$ can be viewed as an element $\mathit{\theta }$ of the manifold $\mathrm{\Theta }$, a function of $L^2(\mathrm{\Theta },\mu )$, $\kappa h(·,\mathit{\theta })$, a function of ${\mathcal{H}}_K$, $K(·,\mathit{\theta })$, or any other equivalent mathematical object, such as composing the function h with a suitable 1-1 smooth function, i.e., $logh(·,\mathit{\theta })$, for example.

For the sake of simplicity, we continue considering that $\mathrm{\Theta }$ as an open subset of ${\mathbb{R}}^m$ and, with some customary abuse of notation, given the local chart $(I,\mathrm{\Theta })$ where I is the identity map, we will identify its points with their coordinates. Under this simplified approach, $\mathit{\theta }=({\theta }^1,\dots ,{\theta }^m)\in \mathrm{\Theta }$ can also be viewed as the coordinates of $\mathit{\varphi }\left(\mathit{\theta }\right)$, a point in the feature manifold. Figure 1 illustrates this framework.

It is also interesting to examine the different representations of the tangent space of $\mathit{\theta }$ in $\mathrm{\Theta }$. Under adequate regularity conditions, an obvious representation consists of considering the span of the functions $T_{\mathit{\theta }}\mathrm{\Theta }\equiv 〈\kappa {\displaystyle \frac{\partial h}{\partial {\theta }^1}}(·,\mathit{\theta }),\dots ,\kappa {\displaystyle \frac{\partial h}{\partial {\theta }^m}}(·,\mathit{\theta })〉$. The Hilbert inner product between them will be given by

$${〈\kappa {\displaystyle \frac{\partial h}{\partial {\theta }^i}}(·,\mathit{\theta }),\kappa {\displaystyle \frac{\partial h}{\partial {\theta }^j}}(·,\mathit{\theta })〉}_{L^2(\mathrm{\Theta },\mu )}={\kappa }^2{\int }_{\mathrm{\Theta }}{\displaystyle \frac{\partial h}{\partial {\theta }^i}}(\mathit{\omega },\mathit{\theta })\overline{{\displaystyle \frac{\partial h}{\partial {\theta }^j}}(\mathit{\omega },\mathit{\theta })}\mu \left(d\mathit{\omega }\right).$$

(29)

Furthermore, with adequate smooth and continuity assumptions, the tangent space will also be identifiable with the space spanned by a linear combination of the partial derivatives of the functions obtained through the canonical feature map, $T_{\mathit{\theta }}\mathrm{\Theta }\equiv 〈{\displaystyle \frac{\partial K}{\partial {\theta }^1}}(·,\mathit{\theta }),\dots ,{\displaystyle \frac{\partial K}{\partial {\theta }^m}}(·,\mathit{\theta })〉$, with the Hilbert inner product between them given by

$${〈{\displaystyle \frac{\partial K}{\partial {\theta }^i}}(·,\mathit{\theta }),{\displaystyle \frac{\partial K}{\partial {\theta }^j}}(·,\mathit{\theta })〉}_{{\mathcal{H}}_K}={\displaystyle \frac{{\partial }^{2}K}{\partial {\theta }^j\partial {\theta }^{m+i}}}(\mathit{\theta },\mathit{\theta })={\displaystyle \frac{{\partial }^{2}K}{\partial {\theta }^{m+j}\partial {\theta }^i}}(\mathit{\theta },\mathit{\theta }).$$

(30)

Notice that, given a smooth map $w:\mathrm{\Theta }\times \mathrm{\Theta }⟶\mathbb{C}$, if we fix a point on the manifold $\mathit{\zeta }\in \mathrm{\Theta }$, then we obtain a function $w_{\mathit{\zeta }}\equiv w(\mathit{\zeta },·)$, which, under regularity conditions, may be approximated through an intrinsic Taylor expansion of rth at $\theta \in \mathrm{\Theta }$, the coefficients of which are the components of convenient tensors—that is, an expansion independent of the coordinate system in $\mathrm{\Theta }$ constructed via the inverse of the Riemannian exponential map corresponding to (24), specifically

$$w_{\mathit{\zeta }}\left(\gamma \right)=w_{\mathit{\zeta }}\left(\theta \right)+\sum _{k=1}^r{\displaystyle \frac{1}{k!}}\left({\nabla }^k{w}_{\mathit{\zeta }}\right)({\exp }_{\mathit{\theta }}^{-1}\left(\mathit{\gamma }\right),\stackrel{k)}{\dots },{\exp }_{\mathit{\theta }}^{-1}\left(\mathit{\gamma }\right))+R_{r,\mathit{\theta }}\left(\mathit{\gamma }\right).$$

(31)

Here, $R_{r,\mathit{\theta }}\left(\mathit{\gamma }\right)$ is their corresponding residual, ${\nabla }^k$ indicates the covariant derivative of k-th order corresponding to the Levi–Civita connection of the Riemannian metric (24), and we have assumed that $\mathit{\gamma }$ belongs to a regular normal neighborhood of $\mathit{\theta }$ to ensure the existence of the inverse of the exponential map of the connection mentioned above, which is always the case if $\mathit{\gamma }$ is close enough to $\mathit{\theta }$.

This development shows that the function $w(\mathit{\gamma },·)$ has different associated covariant tensor fields that can be used in future developments, as well as the contravariant or mixed tensors required by the well-known operation of raising or lowering tensor indices in a Riemannian manifold. Several of the concepts introduced in the present section will be briefly described in Appendix A.

Observe also that, given a second-order mixed tensor field $\widehat{T}$, it corresponds to a linear map operator field, $L_{\widehat{T}}$, defined through $\omega \left(L_{\widehat{T}}\left(\mathsf{{\rm Y}}\right)\right)=\widehat{T}(\omega ,\mathsf{{\rm Y}}),\forall \omega \in T^{\ast }\mathrm{\Theta }\mathrm{and}\forall \mathsf{{\rm Y}}\in T\mathrm{\Theta }$. In any case, we hope to relate these fields of linear operators on the tangent bundle of $\mathrm{\Theta }$, $T\mathrm{\Theta }$, with the classical spectral analysis of these operators in a Hilbert space. In other words, we can consider the appropriate spectral decompositions carried out in each tangent space and eventually interpret them in physical terms, at least as an approximation. Furthermore, through this framework, we can view the statistical estimates of $\mathit{\theta }$, say $\mathit{\gamma }$, as convenient tensor fields in $T\mathrm{\Theta }$.

Additionally, it may be useful to apply the same development to the expected value of these estimations. Given a regular parametric family of functions ${\mathcal{F}}_{\mathrm{\Theta }}$, defined as before, let ${\mathcal{M}}_w:\mathrm{\Theta }⟶\mathbb{C}$ be defined as

$${\mathcal{M}}_w\left(\mathit{\theta }\right)={\int }_{\mathrm{\Theta }}w(\mathit{\gamma },\mathit{\theta })r^2(\mathit{\gamma },\mathit{\theta })\mu \left(d\mathit{\gamma }\right).$$

(32)

As before, under regularity conditions, this map may be approximated through an intrinsic Taylor expansion of the rth at $\theta \in \mathrm{\Theta }$, that is,

$${\mathcal{M}}_w\left(\gamma \right)={\mathcal{M}}_w\left(\theta \right)+\sum _{k=1}^r{\displaystyle \frac{1}{k!}}\left({\nabla }^k{\mathcal{M}}_w\right)({\exp }_{\mathit{\theta }}^{-1}\left(\mathit{\gamma }\right),\stackrel{k)}{\dots },{\exp }_{\mathit{\theta }}^{-1}\left(\mathit{\gamma }\right))+{\mathcal{R}}_{r,\mathit{\theta }}\left(\mathit{\gamma }\right),$$

(33)

where ${\mathcal{R}}_{r,\mathit{\theta }}\left(\mathit{\gamma }\right)$ is their corresponding residual.

2.3.1. The Case of the Gradient Operator

It might sometimes be convenient to consider and represent, in the above framework, real-valued smooth functions defined on $\mathrm{\Theta }$ and their variations throughout the manifold. However, the problem can be stated equivalently in $\mathit{\varphi }\left(\mathrm{\Theta }\right)$ through the canonical feature map, since we assume that the canonical feature map $\mathit{\varphi }$ is 1–1, and therefore, there is a natural identification not only between smooth functions on both manifolds but also between vectors and tensors via the map $\mathcal{T}$, defined as $\mathcal{T}:{\mathbb{R}}^{\mathrm{\Theta }}\to {\mathbb{R}}^{\mathit{\varphi }\left(\mathrm{\Theta }\right)}$, with $f\mapsto \mathcal{T}\left(f\right)=f\circ {\mathit{\varphi }}^{-1}$ and its differential. In other words, we may identify our parameter space $\mathrm{\Theta }$ with its image through the feature map. Figure 1 can help visualize all these considerations.

In this context, it will be convenient to represent a certain function q, for example, $q=w_{\zeta }$ or $q={\mathcal{M}}_w$, in terms of its gradient vector field (see Appendix A for details) since, from a geometric point of view, the gradient of a function indicates the direction of maximum growth of this function per unit of length. Observe that, under the coordinate system $\mathit{\theta }=({\theta }^1,\dots ,{\theta }^m)$, the components of the gradient of q are

$${\mathit{\xi }}^{\alpha }=\mathrm{grad}{\left(q\right)}^{\alpha }\left(\mathit{\theta }\right)=\sum _{\beta =1}^m{g}^{\alpha \beta }\left(\mathit{\theta }\right){\left({\displaystyle \frac{\partial q}{\partial {\theta }^{\beta }}}\right)}_{\mathit{\theta }},\alpha =1,\dots ,m,$$

(34)

where $g^{\alpha \beta }\left(\mathit{\theta }\right)$ are the components of the contravariant metric tensor, i.e., the components of the scalar product in the dual of the tangent space at $\mathit{\theta }$, $T_{\mathit{\theta }}{\mathrm{\Theta }}^{\ast }$, equal to the components of the inverse of the matrix $G={\left(g_{\alpha \beta }\left(\mathit{\theta }\right)\right)}_{m\times m}$ at $\mathit{\theta }$. To simplify the notation, we have identified the points of $\mathrm{\Theta }$ with their coordinates and ${(\partial q/\partial {\theta }^{\beta })}_{\mathit{\theta }}$ with the ordinary partial derivative in ${\mathbb{R}}^m$, $D_{\beta }q\left(\mathit{\theta }\right)$. Notice that in the graphic outputs, in order to describe the behavior of the function q along the entire manifold, it will be enough to represent the field $\mathit{\xi }$ at some specific points of $\mathrm{\Theta }$ of $\mathit{\varphi }\left(\mathrm{\Theta }\right)$: those where it seems most relevant to the applied researcher to determine the direction of growth of q. Observe also that the represented gradient vector at one point will be distorted due to the multidimensional character of the problem, which would have implied some reduction of the dimension, as we will see later. So, it is recommended to calculate the final percentage of the norm of the represented vector. We also note that the growth directions may vary significantly along the manifold.

In a complementary way, it may be interesting to find the curves $\sigma$ corresponding to the integral flow of the gradient, that is, the curves whose tangent vectors at t are ${\sigma }^{\prime }\left(t\right)=\mathrm{grad}\left(q\right)$, which indicate, locally, the maximum variation directions of q per unit of length; see Figure 2. Observe that the trace of these curves orthogonally cuts the level hypersurfaces, a multidimensional extension of the typical contour lines of geographic maps, which we could hypothetically examine. In any case, the posterior representation of these curves in 2D or 3D will be affected by ordinary problems of dimension reduction.

Precisely, under the coordinates $\mathit{\theta }=({\theta }^1,\dots ,{\theta }^m)$, and denoting the curve $\sigma \left(t\right)$ by its coordinates $\mathit{\theta }\left(t\right)$, the integral flow of the gradient of q is the general solution of the first-order differential equation system

$${\displaystyle \frac{d{\theta }^{\alpha }}{dt}}=\sum _{\beta =1}^m{g}^{\alpha \beta }\left(\mathit{\theta }\right){\left({\displaystyle \frac{\partial q}{\partial {\theta }^{\beta }}}\right)}_{\mathit{\theta }},\alpha =1,\dots ,m,$$

(35)

which always has a local solution given initial conditions $\sigma \left(t_0\right)={\mathit{\theta }}_0$, granted by the Picard–Lindelöf theorem; see, for instance [15]. In any case, the curves obtained corresponding to different initial conditions will have to be represented in a low-dimensional Euclidean space, which could potentially generate many interpretation problems, at least if we move away from their corresponding initial points. Therefore, we conclude the following observation:

Although the system (35) could be explicitly solved, at least numerically, the most useful way to represent the function q is by evaluating the gradient vector field of q at some particular points of the manifold. This representation will also facilitate the interpretation of the bi- or three-dimensional Euclidean space of representation.

2.3.2. Projecting onto an Affine Manifold

Consider now the manifold $\mathit{\varphi }\left(\mathrm{\Theta }\right)$ immersed in ${\mathcal{H}}_K$. The tangent space at $\mathit{\varphi }\left(\mathit{\theta }\right)$ can be identified with the affine manifold ${\mathcal{L}}_{\mathit{\theta }}$, which is determined by an m-dimensional subspace spanning ${\mathcal{F}}_{\mathit{\theta }}$ with adequate smoothness and continuity assumptions, by the vectors ${\mathcal{F}}_{\mathit{\theta }}=〈{\displaystyle \frac{\partial K}{\partial {\theta }^1}}(·,\mathit{\theta }),\dots ,{\displaystyle \frac{\partial K}{\partial {\theta }^m}}(·,\mathit{\theta })〉$, with the Hilbert inner product between them given by (30), and a vector $\mathit{\varphi }\left(\mathit{\theta }\right)\in {\mathcal{L}}_{\mathit{\theta }}\subset {\mathcal{H}}_K$, which, in a certain sense, we can take as the origin of ${\mathcal{L}}_{\mathit{\theta }}$. Any vector $\mathit{\eta }\in {\mathcal{L}}_{\mathit{\theta }}\subset {\mathcal{H}}_k$ can be written as $\mathit{\eta }=\mathit{\varphi }\left(\mathit{\theta }\right)+{\sum }_{j=1}^m{\beta }^j{\displaystyle \frac{\partial K}{\partial {\theta }^j}}(·,\mathit{\theta })$. If we introduce matrix notation, defining the $m\times 1$ column vectors $\mathit{\beta }={({\beta }^1,\dots ,{\beta }^m)}^T$ and $\mathit{K}\left(\mathit{\theta }\right)=(\langle \mathit{\eta }-\mathit{\varphi }(\mathit{\theta })$ and ${\displaystyle \frac{\partial K}{\partial {\theta }^1}}(·,\mathit{\theta }){\rangle }_{{\mathcal{H}}_K},\dots ,{\langle \mathit{\eta }-\mathit{\varphi }\left(\mathit{\theta }\right),{\displaystyle \frac{\partial K}{\partial {\theta }^m}}(·,\mathit{\theta })\rangle }_{{\mathcal{H}}_K}{)}^T$ and the $m\times m$ matrix $\mathit{G}\left(\mathit{\theta }\right)=\left({〈{\displaystyle \frac{\partial K}{\partial {\theta }^i}}(·,\mathit{\theta }),{\displaystyle \frac{\partial K}{\partial {\theta }^j}}(·,\mathit{\theta })〉}_{{\mathcal{H}}_K}\right)$, the components of $\mathit{\beta }$ will be given by the following equation:

$$\mathit{\beta }={\mathit{G}}^{-1}\left(\mathit{\theta }\right)\mathit{K}\left(\mathit{\theta }\right).$$

(36)

Additionally, if ${\mathrm{\Pi }}_{{\mathcal{F}}_{\mathit{\theta }}}$ is the projection operator onto ${\mathcal{F}}_{\mathit{\theta }}$, each point in ${\mathcal{H}}_K$ induces a tensor or vector field in $\mathit{\varphi }\left(\mathrm{\Theta }\right)$ via the projection operator field ${\mathrm{\Pi }}_{{\mathcal{F}}_{\mathit{\theta }}}$ that should be deeply connected, at least as an approximation, with the Hermitian linear operators corresponding to convenient observables in quantum mechanics formulation.

2.4. Data Analysis Applications: Mapping Objects in a Low-Dimensional Space

If we are interested in representing data points in the parameter space $\mathrm{\Theta }$ and real functions of ${\mathbb{R}}^{\mathrm{\Theta }}$ in a low-dimensional Euclidean space, usually a plane, we will have two different strategies. In the first, we will require that the final Euclidean distance between points in the plot be, as far as possible, similar to the distance ${\mathcal{H}}_K$ between their images in $\mathit{\varphi }\left(\mathrm{\Theta }\right)$, which shall lead us to a type of kernel PCA method (Principal Component Analysis). In contrast, in the second, we shall require that the final Euclidean distance in the plot be similar to the Riemannian manifold distance in $\mathrm{\Theta }$ of $\mathit{\varphi }\left(\mathrm{\Theta }\right)$ induced by the metric in ${\mathcal{H}}_K$, as in (24). In this case, we shall first map the points in the manifold into a tangent space through the inverse of the exponential map corresponding to the Levi–Civita connection. In this Euclidean space, we shall then use the standard PCA method. Below, we will provide a schematic description of both procedures.

2.4.1. Working in the Feature Space

Among all points we are studying, we may select a specific set of points, which we shall refer to hereafter as principal points, to represent with some optimal property. Specifically, given any set of these principal points ${\mathit{\gamma }}_1,\dots ,{\mathit{\gamma }}_s\in \mathrm{\Theta }$ and given a positive definite Hermitian kernel K in $\mathrm{\Theta }$, we can map them into ${\mathcal{H}}_K$ via the map $\mathit{\varphi }\left(\mathit{\gamma }\right)=K(·,\mathit{\gamma })$, and then we can perform a kernel PCA in the reproducing kernel Hilbert space induced by the kernel, ${\mathcal{H}}_K$. One primary purpose of this analysis is to obtain a low-dimensional Euclidean data representation, such that the distances between the plotted points accurately reflect the differences between the observations. To achieve this, we shall project the points $\mathit{\varphi }\left({\mathit{\gamma }}_i\right)$ onto a convenient r-dimensional linear manifold $\mathcal{L}$ (usually with $r=2$ or 3) immersed in ${\mathcal{H}}_K$. A linear manifold $\mathcal{L}$ is determined by an r-dimensional subspace $\mathcal{F}$ spanned by an orthonormal basis ${\mathit{v}}_1,\dots ,{\mathit{v}}_r\in {\mathcal{H}}_K$ and a vector $\mathit{\xi }\in {\mathcal{H}}_K$ which defines the origin of the data representation. Any vector $\mathit{\eta }\in \mathcal{L}\subset {\mathcal{H}}_K$ can be written as $\mathit{\eta }=\mathit{\xi }+{\sum }_{j=1}^r{\alpha }_j{\mathit{v}}_j,$ and if ${\mathrm{\Pi }}_{\mathcal{F}}$ is the projection operator onto $\mathcal{F}$, each observation $\mathit{\gamma }$ will be represented as a single point in a r-dimensional Cartesian coordinates system, with coordinates ${\alpha }_1,\dots ,{\alpha }_r$, satisfying ${\mathrm{\Pi }}_{\mathcal{F}}(\mathit{\varphi }\left(\mathit{\gamma }\right)-\mathit{\xi })={\sum }_{j=1}^r\langle (\mathit{\varphi }\left(\mathit{\gamma }\right)-\mathit{\xi }),{\mathit{v}}_j\rangle {\mathit{v}}_j={\sum }_{j=1}^r{\alpha }_j{\mathit{v}}_j$. The next step is to find the linear r-dimensional manifold $\mathcal{L}$ following a convenient optimization criterion. Two approaches lead to the same result:

(a) 

To obtain a linear r-dimensional manifold $\mathcal{L}$ immersed in ${\mathcal{H}}_K$, minimize

$$\begin{array}{cc}\hfill \mathcal{D}({\mathit{v}}_1,\dots ,{\mathit{v}}_r,\mathit{\xi })& :=\sum _{j=1}^s{\delta }^2(\mathit{\varphi }\left({\mathit{\gamma }}_j\right),\mathcal{L}),\mathrm{where}\hfill \\ \hfill {\delta }^2(\mathit{\varphi }\left({\mathit{\gamma }}_j\right),\mathcal{L})& =:inf{d}^2(\mathit{\varphi }\left({\mathit{\gamma }}_j\right),\mathbf{y})\hfill \\ & \mathbf{y}\in \mathcal{L}.\hfill \end{array}$$

(37)

Observe that $\mathcal{D}$ is a goodness-of-fit measure. If we minimize this quantity, we shall obtain the linear manifold $\mathcal{L}$ of closest fit to the system of points $\mathit{\varphi }\left({\mathit{\gamma }}_1\right),\dots ,\mathit{\varphi }\left({\mathit{\gamma }}_s\right)$ in the space ${\mathcal{H}}_K$.

(b) 

To obtain a linear r-dimensional manifold $\mathcal{L}$ immersed in ${\mathcal{H}}_K$, maximize

$$\begin{array}{cc}\hfill \Delta ({\mathit{v}}_1,\dots ,{\mathit{v}}_r,\mathit{\xi })& :=\sum _{j=1}^s\sum _{k=j}^s{d}^2({\mathrm{\Pi }}_{\mathcal{F}}(\mathit{\varphi }\left({\mathit{\gamma }}_j\right)-\mathit{\xi }),{\mathrm{\Pi }}_{\mathcal{F}}(\mathit{\varphi }\left({\mathit{\gamma }}_k\right)-\mathit{\xi })).\hfill \end{array}$$

(38)

Notice that $\Delta$ is a measure of the resolving power of the final output. By maximizing this quantity we shall obtain points separated as much as possible in the low-dimensional Euclidean data representation.

Observe also that both $\mathcal{D}$ and $\Delta$ are non-negative real-valued functions with the variables $\mathit{\xi },{\mathit{v}}_1,\dots ,{\mathit{v}}_r$, i.e., the vectors which determine $\mathcal{L}$. Taking into account that

$${\delta }^2(\mathit{\varphi }\left({\mathit{\gamma }}_i\right),\mathcal{L})={\parallel {\mathrm{\Pi }}_{{\mathcal{F}}^{\perp }}(\mathit{\varphi }\left({\mathit{\gamma }}_i\right)-\mathit{\xi })\parallel }^2,$$

(39)

where $\parallel ·\parallel$ is the norm in ${\mathcal{H}}_K$ and ${\mathrm{\Pi }}_{{\mathcal{F}}^{\perp }}$ is the projection operator onto ${\mathcal{F}}^{\perp }$, and defining

$$\overline{\mathit{\varphi }}:={\displaystyle \frac{1}{s}}\sum _{i=1}^s\mathit{\varphi }\left({\mathit{\gamma }}_i\right),$$

(40)

as the average feature, we have

$$\mathcal{D}({\mathit{v}}_1,\dots ,{\mathit{v}}_r,\mathit{\xi })=\mathcal{D}({\mathit{v}}_1,\dots ,{\mathit{v}}_r,\overline{\mathit{\varphi }})+s{\parallel {\mathrm{\Pi }}_{{\mathcal{F}}^{\perp }}(\mathit{\xi }-\overline{\mathit{\varphi }})\parallel }^2.$$

(41)

In order to minimize (a), it is straightforward to prove that we have to choose

$$\mathit{\xi }=\overline{\mathit{\varphi }}:={\displaystyle \frac{1}{s}}\sum _{i=1}^s\mathit{\varphi }\left({\mathit{\gamma }}_i\right).$$

(42)

Let $\mathcal{E}$ be the linear span of $\mathit{\varphi }\left({\mathit{\gamma }}_1\right)-\overline{\mathit{\varphi }},\dots ,\mathit{\varphi }\left({\mathit{\gamma }}_s\right)-\overline{\mathit{\varphi }}$, with $\dim \mathcal{E}=q\le s-1$. If $q\le r$, then the problem has as trivial solutions any $\mathcal{F}$ satisfying $\mathcal{E}\subset \mathcal{F}$, and therefore, $\mathcal{D}=0$. In this case we can choose ${\mathit{v}}_1,\dots ,{\mathit{v}}_r$ in such a way that ${\mathit{v}}_1,\dots ,{\mathit{v}}_q$ so as to be a basis of $\mathcal{E}$. When $q>r$, in order to minimize $\mathcal{D}$, it is clear that we must have $\mathcal{F}⊊\mathcal{E}$, and thus, ${\mathit{v}}_1,\dots ,{\mathit{v}}_r$ is spanned by $\mathit{\varphi }\left({\mathit{\gamma }}_1\right)-\overline{\mathit{\varphi }},\dots ,\mathit{\varphi }\left({\mathit{\gamma }}_s\right)-\overline{\mathit{\varphi }}$; therefore, we can define the $r\times s$ matrix $\mathit{V}$ with elements $v_{li}$ given by

$${\mathit{v}}_i=\sum _{l=1}^s{v}_{li}(\mathit{\varphi }\left({\mathit{\gamma }}_s\right)-\overline{\mathit{\varphi }})i=1,\dots ,r.$$

(43)

Next, we shall define the Gram or kernel matrix corresponding to the kernel k and the sample ${\mathit{\gamma }}_1,\dots ,{\mathit{\gamma }}_s$ as the $s\times s$ matrix $\mathit{K}$, the elements of which are given by $k_{ij}:=\langle \mathit{\varphi }\left({\mathit{\gamma }}_i\right),\mathit{\varphi }\left({\mathit{\gamma }}_j\right)\rangle =$$k({\mathit{\gamma }}_i,{\mathit{\gamma }}_j)$, and the $s\times s$ centering data matrix defined as ${\mathbf{H}}_s:={\mathit{I}}_s-{\displaystyle \frac{1}{s}}{\mathbf{1}}_s{\mathbf{1}}_s^{⊺}$, where ${\mathit{I}}_s$ is the $s\times s$ identity matrix, ${\mathbf{1}}_s$ is an m-dimensional column vector with all components equal to one, and the symbol ⊺ stands for the transpose matrix operator. Then, if we define

$$\overline{\mathit{K}}={\mathit{H}}_s\mathit{K}{\mathit{H}}_s,$$

(44)

where the elements are ${\overline{k}}_{ij}:=\langle \mathit{\varphi }\left({\mathit{\gamma }}_i\right)-\overline{\mathit{\varphi }},\mathit{\varphi }\left({\mathit{\gamma }}_j\right)-\overline{\mathit{\varphi }}\rangle$, we shall have

$$\mathcal{D}=\mathrm{tr}\left(\overline{\mathit{K}}\right)-\mathrm{tr}\left(\overline{\mathit{K}}\mathit{V}{\mathit{V}}^{⊺}\overline{\mathit{K}}\right),$$

(45)

with the orthonormal basis condition, in matrix form, given by

$${\mathbf{V}}^{⊺}\overline{\mathit{K}}\mathit{V}={\mathit{I}}_r.$$

(46)

However, since $\overline{\mathit{K}}$ is a symmetric and positive semidefinite matrix, we can define its symmetric square root ${\overline{\mathit{K}}}^{1/2}$, and then, if we let $\mathit{U}={\overline{\mathit{K}}}^{1/2}\mathit{V}$, we shall have

$$\mathcal{D}=\mathrm{tr}\left(\overline{\mathit{K}}\right)-\mathrm{tr}\left({\overline{\mathit{K}}}^{1/2}\mathit{U}{\mathit{U}}^{⊺}{\overline{\mathit{K}}}^{1/2}\right)=\mathrm{tr}\left(\overline{\mathit{K}}\right)-\mathrm{tr}\left({\mathit{U}}^{⊺}\overline{\mathit{K}}\mathit{U}\right),$$

(47)

with the condition

$${\mathit{U}}^{⊺}\mathit{U}={\mathit{I}}_r.$$

(48)

It is straightforward to prove that $\mathcal{D}$ is minimized if the r columns of $\mathit{U}$, ${\mathit{u}}_1,\dots ,{\mathit{u}}_r$, are the normalized eigenvectors

$$\overline{\mathit{K}}{\mathit{u}}_i={\lambda }_i{\mathit{u}}_i,\mathrm{with}{\lambda }_1\ge {\lambda }_2\ge \dots \ge {\lambda }_r\ge \dots \ge {\lambda }_s=0,$$

(49)

corresponding to the largest eigenvalues of $\overline{\mathit{K}}$. Observe also that

$$\langle \mathit{\varphi }\left({\mathit{\gamma }}_i\right)-\overline{\mathit{\varphi }},{\mathit{v}}_j\rangle =\sum _{l=1}^s{v}_{lj}{\overline{k}}_{il}.$$

(50)

Therefore, in the final output, the Cartesian coordinates of ${\mathit{\gamma }}_i$, $i=1,\dots ,s$, will be given, in matrix notation, by

$$\mathit{A}:={\left(\langle \mathit{\varphi }\left({\mathit{\gamma }}_i\right)-\overline{\mathit{\varphi }},{\mathit{v}}_j\rangle \right)}_{s\times r}=\overline{\mathit{K}}\mathit{V},$$

(51)

or

$$\mathit{A}={\overline{\mathit{K}}}^{1/2}\mathit{U}.$$

(52)

Observe that if the ${\mathit{A}}_i$, $i=1,\dots ,s$, the rows of the matrix $\mathit{A}$ are

$$d^2({\mathrm{\Pi }}_{\mathcal{F}}(\mathit{\varphi }\left({\mathit{\gamma }}_i\right)-\mathit{\xi }),{\mathrm{\Pi }}_{\mathcal{F}}(\mathit{\varphi }\left({\mathit{\gamma }}_j\right)-\mathit{\xi }))=({\mathit{A}}_i-{\mathit{A}}_j){({\mathit{A}}_i-{\mathit{A}}_j)}^{⊺},$$

(53)

and after some straightforward computations, we shall obtain

$$\Delta ({\mathit{v}}_1,\dots ,{\mathit{v}}_r,\mathit{\xi }):=2s\mathrm{tr}\left({\mathit{U}}^{⊺}\overline{\mathit{K}}\mathit{U}\right),$$

(54)

which shows that the criteria (a) and (b) are equivalents.

Notice that we can represent any additional point $\mathit{\eta }\in \mathrm{\Theta }$ in the output plot. Specifically, to find the Cartesian coordinates of the representation of this point $\mathit{\eta }$ in the final output, we compute the projection of the centered $\mathit{\varphi }$-image of $\mathit{\eta }$ into $\mathcal{F}$. For this calculation, notice that

$$〈\mathit{\varphi }\left(\mathit{\eta }\right)-\overline{\mathit{\varphi }},{\mathit{v}}_k〉=\sum _{i=1}^s{v}_{ik}〈\mathit{\varphi }\left(\mathit{\eta }\right)-\overline{\mathit{\varphi }},\mathit{\varphi }\left({\mathit{\gamma }}_i\right)-\overline{\mathit{\varphi }}〉.$$

Introducing the vector

$$\mathit{Z}={\left(k(\mathit{\eta },{\mathit{\gamma }}_i)\right)}_{s\times 1},$$

(55)

from the last expression, we shall have

$${\left(〈\mathit{\varphi }\left(\mathit{\eta }\right)-\overline{\mathit{\varphi }},{\mathit{v}}_k〉\right)}_{1\times r}=\left({\mathit{Z}}^{⊺}-{\displaystyle \frac{1}{s}}{\mathbf{1}}_s^{⊺}\mathit{K}\right){\mathit{H}}_s\mathit{V}.$$

(56)

Notice that Kernel PCA (KPCA) uses only the values of the kernel evaluated at ${\mathit{\gamma }}_1\dots ,{\mathit{\gamma }}_s,\mathit{\eta }$, since the algorithm formulates the reduction of the dimension in the feature space only through the kernel evaluation at these points.

To facilitate the intelligibility of the KPCA output plot, we can represent in it the $\sigma \left(t\right)$ curve’s solution of (35), which indicates, locally, the maximum variation directions of $\widehat{f}$, or alternatively, the corresponding gradient vector field, given in (34).The curve $\sigma \left(t\right)=k(·,\mathit{\theta }(t\left)\right)$, with $\mathit{\theta }\left(t\right)$, is the solution of (35). Then, we can compute the projections of this curve onto the subspace $\mathcal{F}$ spanned by (43). If we define

$${\mathit{Z}}_t={\left(k(\mathit{\theta }\left(t\right),{\mathit{\gamma }}_i)\right)}_{m\times 1},$$

(57)

taking into account Equation (56), the induced curve, $\tilde{\sigma }\left(t\right)$, in $\mathcal{F}$ expressed in matrix form is given by the row vector

$$\tilde{\sigma }{\left(t\right)}_{1\times r}=\left({\mathit{Z}}_t^{⊺}-{\displaystyle \frac{1}{m}}{\mathbf{1}}_m^{⊺}K\right)\left({\mathit{I}}_m-{\displaystyle \frac{1}{m}}{\mathbf{1}}_m{\mathbf{1}}_m^{⊺}\right)\mathit{V},$$

(58)

where ${\mathit{Z}}_t$ is of the form (57).

We can also represent the gradient vector field of $\widehat{f}$, equal to the tangent vector field corresponding to $\sigma \left(t\right)$, through its projection onto the subspace $\mathcal{F}$. The tangent vector at $t=t_0$, if ${\mathit{\theta }}_0={\mathit{\varphi }}^{-1}\circ \sigma \left(t_0\right)$, is given by ${\displaystyle \frac{d\sigma }{dt}}{|}_{t=t_0}$, and its projection, in matrix form, is

$${\displaystyle \frac{d\tilde{\sigma }}{dt}}{|}_{t=t_0}={\displaystyle \frac{d{\mathit{Z}}_t^{⊺}}{dt}}{|}_{t=t_0}\left({\mathbf{I}}_m-{\displaystyle \frac{1}{m}}{\mathbf{1}}_m{\mathbf{1}}_m^{⊺}\right)\mathbf{V},$$

(59)

with

$${\displaystyle \frac{d{\mathit{Z}}_t^{⊺}}{dt}}{|}_{t=t_0}={\left({\displaystyle \frac{d{Z}_t^1}{dt}}{|}_{t=t_0},\dots ,{\displaystyle \frac{d{Z}_t^m}{dt}}{|}_{t=t_0}\right)}^{⊺}.$$

(60)

Taking into account (30), we obtain

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{Z}_t^i}{dt}}{|}_{t=t_0}& ={\displaystyle \frac{dK(\mathit{\gamma }\left(t\right),{\mathit{\gamma }}_i)}{dt}}{|}_{t=t_0}\hfill \\ \hfill & =\sum _{\alpha =1}^m{D}_{\alpha }K({\mathit{\gamma }}_0,{\mathit{\gamma }}_i){\displaystyle \frac{d{x}^{\alpha }}{dt}}{|}_{t=t_0}\hfill \\ \hfill & =\sum _{\alpha =1}^m{D}_{\alpha }K({\mathit{\gamma }}_0,{\mathit{\gamma }}_i)\sum _{\beta =1}^m{g}^{\alpha \beta }\left({\mathit{\gamma }}_0\right)D_{\beta }f\left({\mathit{\gamma }}_0\right)\hfill \\ \hfill & =\sum _{\alpha =1}^m\sum _{\beta =1}^n{D}_{m+\alpha }K({\mathit{\gamma }}_i,{\mathit{\gamma }}_0)g^{\alpha \beta }\left({\mathit{\gamma }}_0\right)D_{\beta }f\left({\mathit{\gamma }}_0\right),\hfill \end{array}$$

(61)

as a result, which basically coincides with [16]. Figure 3 graphically shows the method outlined in the present subsection.

2.4.2. PCA in a Tangent Space

Now, we shall require that the final Euclidean distance in the plot be similar to the Riemannian manifold distance on $\mathrm{\Theta }$; in this case we shall first map the principal points ${\mathit{\gamma }}_1,\dots ,{\mathit{\gamma }}_s\in \mathrm{\Theta }$ onto a convenient tangent space through the inverse of the exponential map corresponding to Levi–Civita connection; see Appendix A for additional details. In this Euclidean space, we shall use the standard PCA method. First, we will fix a point $\mathfrak{C}\in \mathrm{\Theta }$, which we shall refer to as a reference point on which to base the graphical representation. For reasons that will become apparent later, it seems convenient to choose as $\mathfrak{C}$ the Riemannian center of mass of the points ${\mathit{\gamma }}_j,j=1,\dots ,s$, that is, a point $\mathfrak{C}$ such that it is a minimum of the map $\alpha \mapsto \mathcal{H}\left(\mathbf{\alpha }\right)$, where

$$\mathcal{H}\left(\mathbf{\alpha }\right)=\sum _{j=1}^s{\rho }^2(\mathbf{\alpha },{\mathit{\gamma }}_j).$$

(62)

Here, ${\rho }^2(\mathbf{\alpha },{\mathit{\gamma }}_j)$ is the square of the Riemannian distance between the points $\mathbf{\alpha }$ and ${\mathit{\gamma }}_j$. This point is uniquely defined in many interesting applications; among others, see [8,17].

Then, the principal points ${\gamma }_1,\dots ,{\gamma }_s$, and the remaining points that we are interested in plotting ${\gamma }_{s+1},{\gamma }_{s+2},\dots \in \mathrm{\Theta }$ are mapped onto the tangent space $T_{\mathfrak{C}}\mathrm{\Theta }$ through the inverse of the Levi–Civita connection exponential map, ${\exp }_{\mathfrak{C}}^{-1}$, which is well defined (almost everywhere) in most applications. Figure 4 helps us to clarify the role of the inverse of the exponential map. The manifold is the (radius R) basket ball, with the Riemannian metric induced by the Euclidean geometry. See Appendix A for further details.

Principal points play an important role since the computation of the above-mentioned center of mass (if, as we have recommended, it is taken as the reference point $\mathfrak{C}$) and the projector ${\pi }_{\mathcal{L}}$ is only concerned with them, independently of the other studied points in $\mathrm{\Theta }$. The selection of the principal points will depend, of course, on the particular problem, and the possibilities are unlimited. We select as principal points, for example, pairs of points which define the gradient of a particular real-valued function on $\mathrm{\Theta }$, computed at certain fixed points. It would be convenient to select these points interactively with suitable computer software. Therefore, we obtain

$${\mathbf{W}}_1={\exp }_{\mathfrak{C}}^{-1}\left({\gamma }_1\right),\dots ,{\mathbf{W}}_s={\exp }_{\mathfrak{C}}^{-1}\left({\gamma }_s\right),{\mathbf{W}}_{s+1}={\exp }_{\mathfrak{C}}^{-1}\left({\gamma }_{s+1}\right),\dots ,$$

(63)

and, thus, our original points ${\gamma }_1,\dots ,{\gamma }_s,{\gamma }_{s+1},\dots \in \mathrm{\Theta }$ are now represented as points in a m-dimensional Euclidean space $T_{\mathfrak{C}}\mathrm{\Theta }$, with scalar product corresponding to the basis ${(\partial /\partial {\theta }^{\alpha })}_{\mathfrak{C}}$$\alpha =1,\dots ,m$, given by the information matrix at $\mathfrak{C}$, where

$${\left({\displaystyle \frac{\partial }{\partial {\theta }^{\alpha }}}\right)}_{\mathfrak{C}}f=D_{\alpha }(f\circ \mathit{\varphi })\left({\mathit{\varphi }}^{-1}\left(\mathfrak{C}\right)\right)\alpha =1,\dots ,m.$$

(64)

We shall use the same symbol, $\mathit{W}$, to denote the points at $T_{\mathfrak{C}}\mathrm{\Theta }$ and their coordinates with respect to the basis vector field mentioned above, in matrix notation, as a vector $m\times 1$. It is important to note that ${\exp }_{\mathfrak{C}}^{-1}$, considered a map between two metric spaces, preserves the distance between any point and $\mathfrak{C}$, although it does not preserve, in general, distances between arbitrary points, in such a way that the distortion between both distances is small for points close to $\mathfrak{C}$.

More precisely, let $B(\mathfrak{C},R)$ be a Riemannian convex ball of radius $R>0$, and let ${\mathit{\theta }}_{\alpha },{\mathit{\theta }}_{\beta }\in B(\mathfrak{C},R/2)\subset \mathrm{\Theta }$, ${\mathit{\theta }}_{\mathbf{\alpha }}\ne {\mathit{\theta }}_{\beta }$. We shall have

$${\displaystyle \frac{|{\rho }^2({\mathit{\theta }}_{\alpha },{\mathit{\theta }}_{\beta })-|{\exp }_{\mathfrak{C}}^{-1}\left({\mathit{\theta }}_{\alpha }\right)-{\exp }_{\mathfrak{C}}^{-1}\left({\mathit{\theta }}_{\beta }\right){|}_{\mathfrak{C}}^2|}{{\rho }^2({\mathit{\theta }}_{\alpha },{\mathit{\theta }}_{\beta })}}\le {\displaystyle \frac{1}{3}}D{R}^2\{1+O\left(R^2\right)\},$$

(65)

where $D=max\left|\mathcal{K}\right(\mathit{X},\mathit{Y}\left)\right|$ on $\overline{B}(\gamma ,R)$, the closure of the Riemannian ball $\overline{B}(\gamma ,R)$, with $\mathcal{K}(\mathit{X},\mathit{Y})$ being the Riemannian sectional curvature. Observe that the relative error to approximate ${\rho }^2({\mathit{\theta }}_{\alpha },{\mathit{\theta }}_{\beta })$, the square of the Riemannian distance between the points ${\mathit{\theta }}_{\alpha }$ and ${\mathit{\theta }}_{\beta }$, by $|{\exp }_{\mathfrak{C}}^{-1}\left({\mathit{\theta }}_{\alpha }\right)-{\exp }_{\mathfrak{C}}^{-1}\left({\mathit{\theta }}_{\beta }\right){|}_{\mathfrak{C}}^2$ is of the order of $R^2$. See Appendix A for details. To facilitate the reading of the article for the common reader, we have obtained (65), a formula more or less familiar to readers with a background in Riemannian geometry.

This fact suggests the convenience of choosing $\mathfrak{C}$ as the center of a small Riemannian ball that includes the majority of the principal points, with the Riemannian center of mass of the principal points being a good candidate since, in that case, we shall have, lower error bounds for the metric distortions on average. At any rate, it is possible to measure the global distortion introduced through an analogue of the correlation cophenetic coefficient.

Then a usual PCA is performed over the representation of the principal points at $T_{\mathfrak{C}}\mathrm{\Theta }$, ${\mathit{W}}_1,\dots ,{\mathit{W}}_s$, taking into account the scalar product in the tangent space to obtain a q-dimensional affine manifold, $\mathcal{L}$, with $q=2$ or $q=3$, which allows for an optimal representation of the principal points. Since the scalar product matrix at $T_{\mathfrak{C}}\mathrm{\Theta }$, with respect to the vector basis mentioned above, is $\mathbf{G}\left(\mathfrak{C}\right)$, given by (7), the following diagonalization must be solved:

$$\mathit{C}\mathit{u}=\lambda {\mathbf{G}}^{-1}\left(\mathfrak{C}\right)\mathit{u}.$$

(66)

Here, $\mathbf{C}$ is the covariance matrix (or any multiple) of the coordinates of the principal points of representation, ${\mathit{W}}_1,\dots ,{\mathit{W}}_s$, in the tangent space, ${\mathbf{G}}^{-\mathbf{1}}\left(\mathfrak{C}\right)$ is the inverse of the metric tensor matrix at $\mathfrak{C}$, $\lambda$ is an eigenvalue, and $\mathbf{u}$ is a corresponding eigenvector. If $\mathit{U}$ is an $m\times q$ matrix whose columns are the normalized eigenvectors corresponding to the q highest eigenvalues, then the q principal coordinates, in $\mathcal{L}$, of a point $\mathit{W}$, in matrix notation, as a column vector $q\times 1$, are given by ${\mathit{U}}^{⊺}\mathit{W}$. In other words, if ${\pi }_{\mathcal{L}}$ is the projection operator corresponding to the matrix $\mathit{U}$, supplied by PCA, then all statistical objects (not only the principal points) can be represented in $\mathcal{S}$ as

$${\pi }_{\mathcal{L}}\circ {\exp }_{\mathfrak{C}}^{-1}\left({\theta }_1\right),\dots ,{\pi }_{\mathcal{L}}\circ {\exp }_{\mathfrak{C}}^{-1}\left({\theta }_s\right),\dots ,{\pi }_{\mathcal{L}}\circ {\exp }_{\mathfrak{C}}^{-1}\left({\theta }_{s+1}\right),\dots \in \mathcal{L}.$$

(67)

In summary, the whole procedure can be summarized by

$${\mathit{\theta }}_i\stackrel{{\exp }_{\mathfrak{C}}^{-1}}{\to }{\mathit{W}}_i\stackrel{\mathrm{PCA}}{\to }{\mathit{\zeta }}_i={\mathit{U}}^{⊺}{\mathit{W}}_i.$$

(68)

We shall refer hereafter to this procedure as Intrinsic Data Analysis, IDA for short. It is possible to carry out the steps mentioned above, replacing the barycenter $\mathfrak{C}$ by another point ${\mathfrak{C}}^{\prime }$, which may be easier to compute or more suitable in a specific application, although in this case, the metric distortion caused by ${\exp }_{\mathfrak{C}}^{-1}$ will probably increase.

Once the projector ${\pi }_{\mathcal{L}}$ has been computed, each single point on the tangent space can be mapped in $\mathcal{L}$, and the curves $\sigma \left(t\right)=K(·,\mathit{\gamma }(t\left)\right)$ and their tangent vector field can also be mapped on $\mathcal{L}$ in a straightforward way.

2.5. Physical Applications

In the mathematical framework succinctly presented above, we can define the notion of extended information, codified by the estimation $\mathit{\omega }$, obtained from an information source of the true parameter $\mathit{\theta }$, relative to this true value of the parameter, and referred to as an arbitrary point ${\mathit{\theta }}_0$, as

$$\begin{array}{cc}\hfill I_{\mathit{\omega }}\left(\mathit{\theta }\right)& =-\iota \left(logh(\mathit{\omega },\mathit{\theta })-logh(\mathit{\omega },{\mathit{\theta }}_0)\right)\hfill \\ & =-\iota \left(lnr(\mathit{\omega },\mathit{\theta })+i\zeta (\mathit{\omega },\mathit{\theta })-lnr(\mathit{\omega },{\mathit{\theta }}_0)-i\zeta (\mathit{\omega },{\mathit{\theta }}_0)\right),\hfill \end{array}$$

(69)

where log denotes a version of the complex logarithm defined to satisfy $logh(\mathit{\omega },\mathit{\theta })=lnr(\mathit{\omega },\mathit{\theta })+i\zeta (\mathit{\omega },\mathit{\theta })$, with ln being the standard natural logarithm, and $\iota$ being a constant that determines the units of information with which we will work. The implicit dependence of (69) on ${\mathit{\theta }}_0$ is omitted from the notation since its choice will not play any further role when we calculate its gradient in the parametric space. Additionally, (69) will remain invariant under appropriate data changes, and fixed $\mathit{\omega }$ is also invariant under coordinate changes in the parametric manifold, since it is a scalar field on $\mathrm{\Theta }$.

The information provided by a source external to the observer is represented within him, allowing him to increase his understanding of the objects. Although we can consider different levels and types of the said representation, we will focus on two critical aspects: the parameter space with its natural extended information geometry and the observer’s ability to construct plausibility regarding the true value of the parameter, $\mathrm{\Psi }$, in the parameter space $\mathrm{\Theta }$. This estimation is fundamentally a complex square root of a specific kind of subjective conditional probability density with respect the Riemannian volume, induced by the information metric over the parameter space, $\mathrm{\Theta }$, up to a normalization constant. Specifically, we shall write at the beginning

$${\int }_{\mathrm{\Theta }}\mathrm{\Psi }\left(\mathit{\theta }\right)\overline{\mathrm{\Psi }\left(\mathit{\theta }\right)}V\left(d\mathit{\theta }\right)=a>0,$$

(70)

although we shall be particularly interested in the case $a=1$. Observe that in (70), we are integrating with respect to the Riemannian measure defined in (7); therefore, this expression is invariant under coordinate changes. Furthermore, if we intend to define a probability on the parametric manifold, interpretable as a plausibility about the true value of the parameter, we can take the function $\mathrm{\Psi }\overline{\mathrm{\Psi }}$ as the Radon–Nikodym derivative of said probability with respect to the Riemannian volume, also a measure in the same parameter space. Both measures are independent of the coordinate system used and, therefore, $\mathrm{\Psi }\overline{\mathrm{\Psi }}$ will be an invariant scalar field on the parametric manifold $\mathrm{\Theta }$. Then, we can simply define the information encoded by the subjective plausibility $\mathrm{\Psi }$ on the parameter space $\mathrm{\Theta }$relative to the true parameter $\mathit{\theta }$ as

$$\mathrm{\Lambda }\left(\mathit{\theta }\right)=-\iota log\left(\mathrm{\Psi }\left(\mathit{\theta }\right)\right).$$

(71)

This quantity (71) also remains invariant under coordinate changes in the parametric manifold, being another scalar field in $\mathrm{\Theta }$; see also [4].

The Variational Principle

In this context, trusting that many of the abilities of the observer have been efficiently shaped by natural selection in the process of biological evolution, we can propose that the subjective information mentioned above adjusts in some way to the information provided by the source and, in particular, satisfies the following variational principle (see also [4]):

$$\mathrm{\Omega }\left(\mathrm{\Psi }\right)={\int }_{\mathrm{\Theta }}{|\mathrm{grad}{I}_{\mathit{\omega }}\left(\mathit{\theta }\right)-\mathrm{grad}\mathrm{\Lambda }\left(\mathit{\theta }\right)|}^2\mathrm{\Psi }\left(\mathit{\theta }\right)\overline{\mathrm{\Psi }\left(\mathit{\theta }\right)}V\left(d\mathit{\theta }\right).$$

(72)

Here, is a minimum, or at least stationary, and subject to the constraint (70), and assuming that $\mathrm{\Psi }$ and its gradient, $\mathrm{grad}\mathrm{\Psi }$, vanish at the boundary $\partial \mathrm{\Theta }$ or at infinity, which is the case for the model, we have strong reason to believe that the true parameter $\theta$ should belong far from the boundary, clearly inside of $\mathrm{\Theta }$.

Observe that the functional $\mathrm{\Omega }$ is equal to the expected value corresponding to the probability in $\mathrm{\Theta }$ given by the density $\mathrm{\Psi }\overline{\mathrm{\Psi }}$ with respect to the Riemannian volume induced by the metric (24), $V\left(d\mathit{\theta }\right)$, of the square of the norm, corresponding to a complexified vector field in a Riemannian manifold, as indicated in Appendix A. Since the difference between the gradients $\mathrm{grad}{I}_{\mathit{\omega }}\left(\mathit{\theta }\right)$ and $\mathrm{grad}\mathrm{\Lambda }\left(\mathit{\theta }\right)$ is a complexified vector field, i.e., if $\mathrm{grad}{I}_{\mathit{\omega }}\left(\mathit{\theta }\right)-\mathrm{grad}\mathrm{\Lambda }\left(\mathit{\theta }\right)={\mathit{W}}_1\left(\mathit{\theta }\right)+i{\mathit{W}}_2\left(\mathit{\theta }\right)$, then $|\mathrm{grad}{I}_{\mathit{\omega }}{\left(\mathit{\theta }\right)-grad\mathrm{\Lambda }\left(\mathit{\theta }\right)|}^2=|{\mathit{W}}_1{\left(\mathit{\theta }\right)|}^2+{\left|{\mathit{W}}_2\left(\mathit{\theta }\right)\right|}^2$. Observe that (72) is invariant under coordinate changes in $\mathrm{\Theta }$ since the square of the norm inside the integral is invariant and $\mathrm{\Psi }\left(\mathit{\theta }\right)\overline{\mathrm{\Psi }\left(\mathit{\theta }\right)}V\left(d\mathit{\theta }\right)=\mathrm{\Psi }\left(\mathit{\theta }\right)\overline{\mathrm{\Psi }\left(\mathit{\theta }\right)}\sqrt{g\left(\mathit{\theta }\right)}d\mathit{\theta }$ is as well.

Notice also that the source is considered to be objective or at least, strictly speaking, intersubjective, while the parameter space, with its geometric properties, is in some sense built by the observer and is therefore subjective, although strongly conditioned by the source. In future work, some additional restrictions will be added to the variational principle (72) with the aim of more accurately modeling the observation process as a whole.

Any change in the information encoded by $\mathit{\omega }$ caused by considering a change at the source in the parameter space should correspond to a change in the subjective information proposed by the observer. For this reason, we propose that the squared difference of both gradients $\mathrm{grad}{I}_{\mathit{\omega }}$ and $\mathrm{grad}\mathrm{\Lambda }$ would, on average, result in $\mathrm{\Omega }$ being as small as possible, at least locally.

This variational principle is an extension of a previous one presented by the authors in [4], where only regular parametric statistical models with the standard information metric were considered. Based on the former variational principle applied to these models and playing only with basic statistical tools, in particular the information carried by the data in a given statistical model, for instance, see [18,19,20], we obtain a probability density in the parameter space $\mathrm{\Theta }$ by solving a system of partial differential equations. This probability can be viewed as a Bayesian posterior probability over all possible probabilistic mechanisms that have generated the data, probabilistic mechanisms identifiable with the parameters. Furthermore, if we apply this procedure of analyzing data to the simple statistical model corresponding to a multivariate normal distribution with a constant covariance matrix, a model that can be considered at least as an approximation of many regular statistical models for large samples via extensions of the Central Limit Theorem, we obtain, from those above-mentioned partial differential equations in the parameter space, a specific differential equation already studied in Physics and known as the stationary (time-independent) Schrödinger equation applied to a quantum harmonic oscillator.

We will soon present another work where the variational principle (72) will be resolved with, at least, the restriction (70).

3. Discussion

Understanding the complex relationship between data analysis and physics necessitates a comprehensive examination of the foundational elements that underpin observation and modeling. This paper emphasizes the intrinsic connection between information sources—conceptualized as entities providing sequences of data—and the physical models that aim to represent reality with causal nuance. Central to this discourse is the role of observation, which is depicted as a triadic interaction involving the studied object, the knowing subject, and the environment.

By dissecting these elements within a mathematical framework, this work bridges the cognitive process of observation with the geometric and probabilistic structures that characterize physical models. The observation process, as articulated, hinges on the recognition that all relevant phenomena are ultimately situated within the observer’s consciousness. However, simplifying assumptions facilitates the development of a formal model, where entities involved in observation—objects, subjects, and environments—are represented through elements of a parameter space $\mathrm{\Theta }$. This space, modeled as a smooth manifold, encapsulates the parameters that characterize the underlying data-generating processes.

Such a manifold structure allows for the application of differential geometry, enabling the exploration of the intrinsic properties of models through tools like the Riemannian metric induced by divergence functions or distance measures. The introduction of complex-valued functions $h(\omega ,\theta )$ and their exponential representations illustrates how models of data sources can be embedded within a Hilbert space framework. By defining a family of functions $F_{\mathrm{\Theta }}$, we leverage the geometric interpretation of statistical models, where distances between models—quantified via functions such as the Hilbert metric $D_2$ or the Hellinger distance $d_H$—are instrumental in understanding model complexity and distinguishability.

The construction of a Riemannian metric from these divergence measures aligns with the principles of information geometry, providing a natural geometric framework for analyzing the parameter space. Specifically, the metric tensor components are derived from the second derivatives of divergence functions, which encapsulate the local curvature of the parameter space. This curvature, reflecting the model’s complexity and sensitivity to parameter variations, influences estimation procedures and inference accuracy.

The Fisher information matrix emerges as a pivotal component, quantifying the amount of information that an observable data source contains about the parameters. When combined with phase-related terms arising from complex exponentials (e.g., the functions $\zeta (\omega ,\theta )$), the metric captures both probabilistic and phase information—concepts reminiscent of quantum information geometry. The extension to quantum contexts underscores the universality of such geometric structures. Quantum states, represented by vectors in complex Hilbert spaces, inherently embody both amplitude and phase.

The metric structures derived from complex functions thus encode not only classical probabilistic information—determinable via the Fisher information—but also quantum phases that influence phenomena such as interference and entanglement. This dual encoding exemplifies how geometric tools can unify seemingly disparate layers of information, providing a comprehensive framework for understanding complex systems. This underscores the profound role of geometric and probabilistic structures in modeling physical phenomena and data sources.

Formalizing observation through parameter spaces endowed with Riemannian metrics derived from divergence measures provides a versatile toolkit applicable to both classical and quantum domains. The approach highlights that models are not merely static representations but dynamic entities with curvature and structure that influence estimation, inference, and ultimately, our understanding of reality. Integrating these geometric insights into physical modeling enhances both the conceptual framework and the practical methodologies employed in analyzing complex systems, thereby facilitating more nuanced and precise scientific investigations.

We present a method for effectively connecting two distinct ways of representing data through a specialized mathematical transformation, referred to as a “feature map.” When this feature map is one-to-one—ensuring that each point in the original space corresponds uniquely to a single point in the transformed space—we can establish a reliable method for transferring various functions and objects from the original domain to the new one.

This approach is further enriched by the kernel method, which allows us to implicitly work within a high-dimensional feature space without explicitly computing the feature map. Specifically, kernel functions enable us to compute inner products in the transformed space directly from the original data, facilitating efficient computations even in infinite-dimensional feature spaces. This capability ensures that mathematical entities, such as functions, vectors, and matrices, which characterize the data, can be consistently interpreted within this new context, preserving all information and structural properties.

Moreover, when considering data residing on Riemannian manifolds—smooth, curved spaces that generalize Euclidean geometry—these ideas become even more powerful. Riemannian manifolds provide a natural setting for modeling complex data that intrinsically possesses geometric structure, such as shapes, surfaces, or biological structures. In such contexts, the notion of a feature map can be extended to respect the manifold’s geometric properties, allowing for transformations that preserve curvature and intrinsic distances.

By leveraging kernels adapted to Riemannian manifolds—so-called Riemannian kernels—we can perform implicit high-dimensional embedding that accounts for the manifold’s geometry, enabling meaningful comparisons and analyses of data points lying on curved spaces. This synergy between kernel methods and Riemannian geometry facilitates a more profound understanding of complex data structures, supporting tasks such as classification, regression, and visualization within intrinsically curved spaces. Overall, the integration of Riemannian manifolds into kernel-based feature mappings broadens the scope of data representation, ensuring that the geometric essence of the data is maintained throughout the transformation process.

Then, we present a sophisticated approach to analyzing data on Riemannian manifolds, combining differential geometry, kernel methods, and principal component analysis (PCA). At its core, the methodology emphasizes understanding the intrinsic geometric structure of the data, which is crucial when the data naturally resides in nonlinear, curved spaces rather than Euclidean ones. This approach is particularly relevant in fields where the data’s inherent geometry cannot be ignored.

One of the central themes revolves around the flow lines of the gradient vector field, denoted as curves $\sigma \left(t\right)$, which are solutions to a first-order differential system. These curves represent the integral flow of the gradient of a function q, effectively indicating the directions of maximum variation of q on the manifold. Geometrically, these curves intersect the level hypersurfaces orthogonally, much like contour lines on a geographic map, thereby providing a visual and intuitive understanding of the function’s local behavior. The analysis of these flow lines is essential in understanding the local structure of the data and in identifying directions that capture the most significant variations, which can guide dimension reduction strategies.

The representation of these flow lines, especially in low-dimensional projections, is complicated by the manifold’s nonlinear nature. We discuss the challenges associated with dimension reduction, highlighting the potential distortion when projecting high-dimensional manifold data into Euclidean spaces. This issue is addressed through the use of the exponential and logarithmic maps, which transfer points between the manifold and its tangent spaces, preserving local geometric properties. The inverse exponential map, ${\exp }^{-1}$, maps points from the manifold to the tangent space at a reference point, facilitating the application of Euclidean PCA in the tangent space. The choice of the Riemannian center of mass as the reference point is significant, as it ensures that the PCA captures the dominant variation directions in a manner faithful to the manifold’s geometry.

Kernel methods, particularly Kernel PCA (KPCA), are integrated into this framework to perform nonlinear dimension reduction in the feature space induced by a kernel function. The kernel encapsulates the manifold’s intrinsic geometry implicitly, allowing for the analysis of data that may be difficult to embed linearly. We describe how the kernel matrix K, constructed from evaluations of the kernel function at data points, encodes geometric relationships, such as distances and angles, in a high-dimensional feature space. This means that the kernel matrix captures the relationships between data points in a way that respects the manifold’s geometry.

Eigendecomposition of the kernel matrix reveals principal directions of variation, which are then projected back into the original data space. This process enables the visualization of complex structures, such as flow lines and gradient vector fields, in low-dimensional representations that respect the manifold’s intrinsic distances. Furthermore, the methodology incorporates a nuanced treatment of the curvature effects inherent in Riemannian manifolds. The approximation of the squared distances using the inverse exponential map, with bounds related to sectional curvature, ensures that local linearizations remain valid within convex neighborhoods. This means that within these neighborhoods, the manifold can be approximated as a Euclidean space, allowing for the application of linear methods such as PCA.

By selecting a central point, such as the Riemannian barycenter, the analysis minimizes distortion, resulting in more faithful low-dimensional visualizations. The tangent space PCA—referred to as intrinsic PCA—operates on the mapped principal points, allowing for an eigendecomposition that respects the metric tensor, thus accounting for the manifold’s local curvature. This combined geometric and statistical framework underscores the importance of respecting the data’s underlying structure. By integrating differential geometric concepts with kernel methods and PCA, the approach achieves a balance between local fidelity and global interpretability. It enables the extraction of meaningful features, flow patterns, and principal directions that are invariant under the manifold’s curvature, resulting in more robust and insightful analyses.

This methodology exemplifies a comprehensive approach to manifold data analysis, emphasizing the importance of intrinsic geometry. The use of flow lines of the gradient, the application of exponential and logarithmic maps, the implementation of kernel-based reduction, and the intrinsic PCA all serve to capture the complex structure of data residing in nonlinear spaces. Such techniques are increasingly vital in modern data science, where high-dimensional, curved data structures are prevalent, and a faithful representation of their geometry is essential for meaningful interpretation.

Building on this synthesis, the geometric framework we propose opens new avenues for understanding the dynamics of information in physical systems. By conceptualizing inference processes as variational principles operating on Riemannian manifolds, we suggest that the evolution of knowledge—both in scientific models and in fundamental physics—may be described by wave-like phenomena governed by invariant geometric structures. This perspective aligns with the notion that physical laws themselves can be viewed as extremal principles on curved manifolds, where the flow of information and the evolution of states are intrinsically linked. Such a viewpoint encourages us to explore whether the principles underlying quantum mechanics, general relativity, and thermodynamics can be unified within a common geometric language rooted in information theory.

Consequently, this approach not only enriches our theoretical understanding of inference but also suggests a deeper, possibly holographic, nature of reality, where the fabric of spacetime and the behavior of matter emerge from the geometry of information. Future research may reveal whether these geometric and variational insights can lead to new formulations of physical laws, bridging the gap between abstract information geometry and the tangible fabric of the universe.

Of course, this paper is only the first step towards the above-mentioned deep relationship between physical and data analysis. We would certainly like to have a clear path already outlined for developing the variational principle and connecting it with the major existing physical theories. For the time being, in our immediate work, we will limit ourselves to attempting to derive some existing basic equations—the Schrödinger, Klein-Gordon, and Dirac equations—and to the extent possible, we will explore their compatibility with general relativity and thermodynamics. Other relationships, such as decoherence, entanglement, and von Neumann’s entropy, for example, will be addressed in future work.