An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory

Abstract

Shannon’s differential entropy for continuous variables suffers from a fundamental limitation: it is not invariant under scale transformations. This makes entropy values dependent on the choice of measurement units rather than reflecting intrinsic properties of distributions. While Jaynes proposed the limiting density of discrete points (LDDP) as a theoretical solution, a concrete method for computing the required invariant measure has been lacking. This paper establishes a rigorous connection between Kullback–Leibler divergence and the invariant measure, providing theoretical proofs of invariance under affine transformations and a practical computational method. We prove that entropy normalized by the median of k-nearest neighbor distances is invariant under affine transformations (Theorems 1 and 2). The non-negativity of the resulting entropy has been validated empirically across all tested distribution families, though a complete theoretical proof remains an open question. This approach extends naturally to multivariate settings, enabling scale-invariant mutual information estimation. We provide open-source implementations in Julia (EntropyInvariant.jl) and Python (entropy_invariant) and demonstrate their advantages over traditional approaches, particularly for variables with disparate scales.

1. Introduction

Information theory provides powerful tools for quantifying relationships between variables across scientific disciplines. Among these, mutual information (MI) stands out for its ability to capture both linear and nonlinear dependencies while remaining robust to small sample sizes [1,2,3]. Unlike correlation-based measures, MI is sensitive to the complete dependence structure between variables, making it particularly valuable for complex data analysis [4].

The foundation of MI lies in Shannon’s entropy framework [5], originally defined for discrete variables and later extended to continuous variables as differential entropy. For a continuous random variable X with probability density function ${\mu }_X$, the differential entropy is:

$$h\left(X\right)=-{\int }_X{\mu }_X\left(x\right)·\log \left({\mu }_X\left(x\right)\right)·\mathrm{d}x$$

(1)

This definition applies to both bounded and unbounded domains, provided the integral converges. Our invariant measure methodology requires that the median of k-nearest neighbor distances stabilizes with increasing sample size, a condition empirically satisfied even for heavy-tailed distributions such as Cauchy and Lévy (Table 2).

However, differential entropy suffers from a critical limitation: it is not invariant under scale transformations. Under a linear transformation $Y=aX$, the entropy transforms as:

$$h\left(aX\right)=h\left(X\right)+\log \left(\right|a\left|\right)$$

(2)

This scale-dependence means that entropy values depend on measurement units rather than reflecting intrinsic properties of the distribution. For instance, measuring temperature in Celsius versus Fahrenheit yields different entropy values, even though the underlying physical system is identical. This limitation extends to mutual information, potentially leading to misleading results when comparing variables with different scales.

MI is zero when X and Y are independent and increases with the dependence between variables. This measure is always non-negative and has no upper bound, expressed as $MI(X,Y)\ge 0$. MI can be expressed in terms of differential entropy as:

$$MI(X;Y)=h\left(X\right)+h\left(Y\right)-h(X,Y)$$

(3)

where $h(X,Y)$ denotes the joint differential entropy. Equivalently, $MI(X;Y)=D_{KL}(P_{X,Y}\parallel P_X\otimes P_Y)$, measuring the KL divergence from the joint distribution to the product of marginals. While MI shares some metric-like properties (non-negativity, symmetry), it does not satisfy the triangle inequality and is therefore not a true metric in the mathematical sense; we use the term “measure of dependence” throughout.

A significant challenge in many practical applications lies in estimating entropy from finite samples when the underlying probability density function (pdf) is unknown. Techniques for entropy estimation can be broadly classified into two main categories: parametric and nonparametric methods [6]. Parametric methods assume that the form of the pdf is known, reducing the problem to estimating the parameters from the data [1]. Nonparametric methods do not make assumptions about the form of the pdf, making them more versatile and widely applicable. Among these, popular approaches include histogram-based methods [7], kernel density estimators (KDEs) [8], and entropy-based statistical tests [9]. Another nonparametric approach is based on the k-nearest neighbors (kNN) method [3,6], which is computationally efficient and has been shown to be robust even with small sample sizes [10,11].

Despite their popularity, traditional nonparametric methods have significant limitations. Histogram-based methods are highly sensitive to the choice of bin width: too few bins lead to oversmoothing and information loss, while too many bins result in high variance and systematic bias that increases with the number of bins. The optimal bin width is data-dependent and difficult to determine a priori. KDE methods face similar bandwidth selection challenges. While the kNN method addresses some of these issues by using a single, interpretable parameter k and demonstrating good convergence properties, it remains sensitive to scale transformations. For comprehensive coverage of information-theoretic foundations, we refer readers to [12].

The root of the scale-dependence problem lies in the transition from discrete to continuous entropy. To understand this, consider the Kullback–Leibler (KL) divergence [13], which measures the dissimilarity between distributions P and Q:

$$D_{KL}\left(P\right|\left|Q\right)={\int }_X{\mu }_P\left(x\right)·\log \left({\displaystyle \frac{{\mu }_P\left(x\right)}{{\mu }_Q\left(x\right)}}\right)·\mathrm{d}x$$

(4)

The KL divergence relates to differential entropy through:

$$D_{KL}\left(P\right|\left|Q\right)=-h\left(P\right)-{\mathbb{E}}_P\left[\log \left({\mu }_Q\left(X\right)\right)\right]$$

(5)

When Q is a uniform distribution over an interval of length r, we obtain:

$$h\left(P\right)=\log \left(r\right)-D_{KL}\left(P\right||U_r)$$

(6)

This reveals that differential entropy implicitly compares the distribution to a uniform reference whose scale affects the entropy value. The scale-dependence arises because the reference scale r changes under transformations.

Jaynes [14] recognized this fundamental issue and proposed addressing it through an “invariant measure” $m\left(x\right)$ representing complete ignorance about the probability distribution. This leads to the limiting density of discrete points (LDDP):

$$H_c\left(X\right)=-{\int }_X{\mu }_X\left(x\right)·\log \left({\displaystyle \frac{{\mu }_X\left(x\right)}{m\left(x\right)}}\right)·\mathrm{d}x$$

(7)

Comparing Equations (4) and (7), we observe that the LDDP can be interpreted as a KL divergence where the reference distribution is replaced by the invariant measure. However, Jaynes did not provide a concrete method for computing $m\left(x\right)$, leaving the central question unanswered: what properties must $m\left(x\right)$ satisfy, and how can it be estimated from data?

Recent work by Nagel and coworkers [15] proposed normalizing MI using an invariant measure, but their approach introduces inconsistencies: the marginal entropies vary depending on which variables are paired together, violating the fundamental principle that a variable’s entropy should be intrinsic to that variable alone.

Our contributions: This paper establishes a rigorous connection between the KL divergence framework and Jaynes’s invariant measure concept. We demonstrate that:

• The invariant measure $m\left(x\right)$ can be rigorously defined through specific mathematical properties and computed from data using k-nearest neighbor distances;
• This measure naturally emerges from requiring transformation invariance analogous to the KL divergence framework;
• The resulting invariant entropy corresponds to the LDDP and is truly scale-invariant;
• The approach generalizes naturally to multivariate settings, enabling consistent scale-invariant MI estimation;
• The median of nearest neighbor distances provides a robust estimator that avoids negative entropy values and identifies distribution families.

The present manuscript is organized as follows. Section 2 develops the theoretical framework connecting KL divergence to the invariant measure, proves invariance properties, and presents the computational method. Section 3 validates the approach through simulations and demonstrates its advantages over traditional methods. Section 4 discusses connections to information geometry, maximum entropy principles, and broader implications. Section 5 concludes with future directions.

2. Materials and Methods

2.1. From Kullback–Leibler Divergence to Invariant Measure

To understand the connection between KL divergence and invariant entropy, we examine what happens when comparing a distribution to a uniform reference. We use a bounded interval here for pedagogical clarity; the resulting invariant measure framework applies broadly to distributions with bounded or unbounded support, as demonstrated in Section 3. Consider a uniform distribution $U_{[a,b]}$ with density ${\mu }_U\left(x\right)=1/(b-a)$ for $x\in [a,b]$.

For a distribution P with support contained in $[a,b]$, the KL divergence is:

$$D_{KL}\left(P\right||U_{[a,b]})=-h\left(P\right)+\log (b-a)$$

(8)

Under an affine transformation $Y=aX+b$, the uniform reference transforms to $U_{[aa+b,ab+b]}$ with density $1/\left(\right|a\left|\right(b-a\left)\right)$. The KL divergence becomes:

$$D_{KL}(P_Y\left|\right|U_Y)=-h\left(X\right)+\log (b-a)=D_{KL}(P_X\left|\right|U_X)$$

(9)

This demonstrates that the KL divergence to a uniform reference is invariant under affine transformations, but differential entropy is not because the log-volume term changes. The key insight is that we need a reference measure that adapts to the data scale in a way that removes this scale-dependence.

2.2. Definition of the Invariant Measure

We propose that the invariant measure $m\left(x\right)$ should satisfy properties that ensure scale invariance while remaining practically computable from data.

Proposition 1.

Let $m:\mathbb{R}\to {\mathbb{R}}^{+}$ be an invariant measure function. We require:

(i)    

Positivity: $m\left(X\right)=r_X>0$ for any random variable X;

(ii)   

Scale equivariance: $m\left(aX\right)=\left|a\right|·m\left(X\right)$ for any $a\ne 0$;

(iii)  

Translation invariance: $m(X+b)=m\left(X\right)$ for any $b\in \mathbb{R}$;

(iv)  

Consistency with KL divergence: The measure should lead to an entropy-like quantity that behaves as a KL divergence from the data distribution to a reference distribution.

These properties ensure that $m\left(X\right)$ captures the intrinsic scale of the distribution independent of affine transformations. We now define the invariant differential entropy as:

$$h_c\left(X\right):=h\left({\displaystyle \frac{X}{m\left(X\right)}}\right)$$

(10)

Theorem 1

(Invariance of $h_c$). Let $m\left(X\right)$ satisfy Properties (i)–(iii) of Proposition 1. Then, for any transformation $Y=aX+b$ with $a\ne 0$:

$$h_c\left(Y\right)=h_c\left(X\right)$$

(11)

Proof. 

Let $Y=aX+b$. Based on Properties (iii) and (ii), we have:

$$m(aX+b)=m\left(aX\right)=\left|a\right|·m\left(X\right)$$

(12)

Therefore:

$$h_c\left(Y\right)=h\left({\displaystyle \frac{Y}{m\left(Y\right)}}\right)=h\left({\displaystyle \frac{aX+b}{\left|a\right|·m\left(X\right)}}\right)$$

(13)

Let $Z=X/m\left(X\right)$ be the normalized variable. Under the transformation $Y=aX+b$, we have:

$${\displaystyle \frac{Y}{m\left(Y\right)}}={\displaystyle \frac{aX+b}{\left|a\right|·m\left(X\right)}}=\mathrm{sgn}\left(a\right)·{\displaystyle \frac{X}{m\left(X\right)}}+{\displaystyle \frac{b}{\left|a\right|·m\left(X\right)}}=\mathrm{sgn}\left(a\right)·Z+c$$

(14)

where $c=b/\left(\right|a|·m(X\left)\right)$ is a constant and $\mathrm{sgn}\left(a\right)=\pm 1$.

Now, we apply the change-of-variables formula for differential entropy. For a random variable Z with density ${\mu }_Z\left(z\right)$ and a transformation $W=g\left(Z\right)$ where g is differentiable and invertible, the density of W is:

$${\mu }_W\left(w\right)={\mu }_Z\left(g^{-1}\left(w\right)\right)·\left|{\displaystyle \frac{d{g}^{-1}}{dw}}\right|$$

(15)

For a linear transformation $W=aZ+c$, we have $g^{-1}\left(w\right)=(w-c)/a$ and $|d{g}^{-1}/dw|=1/|a|$, giving:

$$h(aZ+c)=-\int {\mu }_W\left(w\right)\log \left({\mu }_W\left(w\right)\right)dw=-\int {\displaystyle \frac{1}{\left|a\right|}}{\mu }_Z\left({\displaystyle \frac{w-c}{a}}\right)\log \left({\displaystyle \frac{1}{\left|a\right|}}{\mu }_Z\left({\displaystyle \frac{w-c}{a}}\right)\right)dw$$

(16)

Substituting $u=(w-c)/a$, $dw=\left|a\right|du$:

$$h(aZ+c)=-\int {\mu }_Z\left(u\right)\log \left({\displaystyle \frac{1}{\left|a\right|}}{\mu }_Z\left(u\right)\right)\left|a\right|{\displaystyle \frac{du}{\left|a\right|}}=-\int {\mu }_Z\left(u\right)[\log \left({\mu }_Z\left(u\right)\right)-\log \left(\right|a\left|\right)]du$$

(17)

$$=-\int {\mu }_Z\left(u\right)\log \left({\mu }_Z\left(u\right)\right)du+\log \left(\right|a\left|\right)\int {\mu }_Z\left(u\right)du=h\left(Z\right)+\log \left(\right|a\left|\right)$$

(18)

However, for our normalized variable, $a=\mathrm{sgn}\left(a\right)$ has $\left|a\right|=1$, so:

$$h\left(\mathrm{sgn}\left(a\right)·Z+c\right)=h\left(Z\right)+\log \left(1\right)=h\left(Z\right)=h\left({\displaystyle \frac{X}{m\left(X\right)}}\right)=h_c\left(X\right)$$

(19)

Therefore, $h_c\left(Y\right)=h_c\left(X\right)$, establishing invariance.    □

2.3. Connection to Jaynes’s LDDP

We now show that $h_c\left(X\right)$ is equivalent to Jaynes’s LDDP. Using the change of variables $u=x/m\left(X\right)$, we have $x=m\left(X\right)·u$ and $dx=m\left(X\right)·du$. The density transforms as:

$${\mu }_{X/m\left(X\right)}\left(u\right)=m\left(X\right)·{\mu }_X(m\left(X\right)·u)$$

(20)

Therefore:

$$\begin{array}{cc}\hfill h_c\left(X\right)& =h\left({\displaystyle \frac{X}{m\left(X\right)}}\right)=-\int {\mu }_{X/m\left(X\right)}\left(u\right)\log \left({\mu }_{X/m\left(X\right)}\left(u\right)\right)du\hfill \\ \hfill & =-\int m\left(X\right){\mu }_X\left(m\left(X\right)u\right)\log \left(m\left(X\right){\mu }_X\left(m\left(X\right)u\right)\right)du\hfill \\ \hfill & =-\int {\mu }_X\left(x\right)\log \left(m\left(X\right){\mu }_X\left(x\right)\right){\displaystyle \frac{dx}{m\left(X\right)}}\hfill \\ \hfill & =-\int {\mu }_X\left(x\right)[\log \left({\mu }_X\left(x\right)\right)+\log \left(m\left(X\right)\right)]dx\hfill \\ \hfill & =h\left(X\right)-\log \left(m\right(X\left)\right)\hfill \end{array}$$

(21)

This can be expressed as:

$$h_c\left(X\right)=-\int {\mu }_X\left(x\right)\log \left({\displaystyle \frac{{\mu }_X\left(x\right)}{1/m\left(X\right)}}\right)dx$$

(22)

which is precisely Jaynes’s expression (7) with constant invariant measure $m\left(x\right)=1/m\left(X\right)$. Hence, as mentioned in the Introduction, the invariant entropy corresponds to comparing the distribution to a uniform reference over an interval of length $m\left(X\right)$, which adapts to the data scale.

2.4. Estimation of the Invariant Measure

To make this framework practical, we need a method to estimate $m\left(X\right)$ from data. We propose using the k-nearest neighbor (kNN) approach, which aligns naturally with kNN-based entropy estimation methods.

Given a sample $\{x_1,x_2,\dots ,x_n\}$ from X, we compute the distance from each point to its k-th nearest neighbor:

$$d_i^{\left(k\right)}=\underset{j\ne i}{\overset{\left(k\right)}{\min }}|x_i-x_j|,i=1,\dots ,n$$

(23)

where ${\min }^{\left(k\right)}$ denotes the k-th smallest value. For simplicity, we focus on $k=1$ (nearest neighbor). The vector of nearest neighbor distances $\mathbf{d}=(d_1,\dots ,d_n)$ captures the local density structure. We propose:

$$m\left(X\right)=\mathrm{median}\left(\mathbf{d}\right)$$

(24)

Justification for using the median:

(1) Robustness: The median is robust to outliers, reflecting “complete ignorance” about points far from the data bulk. Consider $X=\{2,3,5,7,11,17,19,23,29\}$ with nearest neighbor vector $\mathbf{d}=\{1,1,2,2,4,2,2,4,6\}$ (sorted: $\{1,1,2,2,2,2,4,4,6\}$), giving $m\left(X\right)=2$ and $\mathrm{mean}\left(\mathbf{d}\right)=2.67$. Adding an outlier $x_{10}=100$ yields $d_{10}=|100-29|=71$ and ${\mathbf{d}}^{\prime }=\{1,1,2,2,4,2,2,4,6,71\}$, still giving $m\left(X^{\prime }\right)=2$. The mean would change from $2.67$ to $9.50$, demonstrating inferior robustness.

(2) Scale equivariance: For scaled data $aX$, nearest neighbor distances scale as $\left|a\right|d_i$, so median$\left(\right|a\left|\mathbf{d}\right)=\left|a\right|$ median$\left(\mathbf{d}\right)$, satisfying Property (ii) of Proposition 1.

(3) Translation invariance: Translating data by b does not change distances, so median$\left(\mathbf{d}\right)$ remains unchanged, satisfying Property (iii).

(4) Avoids negative entropy: Using the mean can lead to negative entropy. For example, the exponential and normal distributions yield negative values with the mean-based measure (

Table 1. Comparison of entropy values using mean versus median invariant measure for exponential and normal distributions. We use standard parameterizations: $\mathcal{E}\left(\lambda \right)$ with density $\lambda e^{-\lambda x}$ for $x\ge 0$ (mean $=1/\lambda$); $\mathcal{N}(\mu ,\sigma )$ with density ${\displaystyle \frac{1}{\sigma \sqrt{2\pi }}}{e}^{-{(x-\mu )}^2/\left(2{\sigma }^2\right)}$. Numerical values use $\lambda =1$ and $\mathcal{N}(0,1)$.

Entropy (Mean)	Entropy (Median)
$h_{c_{\mathrm{mean}}}\left(\mathcal{E}\left(\lambda \right)\right)=-0.06$	$h_{c_{\mathrm{median}}}\left(\mathcal{E}\left(\lambda \right)\right)=1.22$
$h_{c_{\mathrm{mean}}}\left(\mathcal{N}(\mu ,\sigma )\right)=-0.80$	$h_{c_{\mathrm{median}}}\left(\mathcal{N}(\mu ,\sigma )\right)=1.14$

):

Empirically, the median avoids negative entropy values while preserving the desired invariance properties. This robustness likely stems from the median’s optimality as an $L^1$ center, which is less sensitive to extreme values in the distance distribution than the mean ($L^2$ center). A rigorous proof establishing conditions under which the median measure guarantees non-negative entropy for all distribution classes remains an important open theoretical question.

Theoretical status: Our main results (Theorems 1 and 2) rigorously establish that the median-based invariant measure satisfies scale equivariance and translation invariance. The non-negativity of $h_c\left(X\right)$ has been verified empirically for all distribution families in

Table 2. Invariant entropy for common distributions. Each value represents the mean ± standard deviation over 100 simulations with 10,000 samples each. For bounded distributions: Arcsine$(0,1)$, Uniform$(0,1)$, Semicircle with $r=1$, Triangular$(0,1,0.5)$, Cosine$(\mu =0.5,s=1)$. For unbounded distributions: standard parameters were used (e.g., $\mathcal{N}(0,1)$, $\mathcal{E}\left(1\right)$). Due to the invariance property, these values remain constant for any choice of location and scale parameters within each distribution family.

Distribution	Invariant Entropy
Arcsine $(a,b)$	1.008 ± 0.006
Uniform $(a,b)$	1.060 ± 0.005
Semicircle $\left(r\right)$	1.073 ± 0.005
Triangular $(\mu ,\sigma )$	1.106 ± 0.005
Cosine $(\mu ,\sigma )$	1.114 ± 0.005
Normal $(\mu ,\sigma )$	1.150 ± 0.005
Rayleigh $\left(\sigma \right)$	1.135 ± 0.005
Chi $\left(\nu \right)$	1.149 ± 0.005
Gumbel $(\mu ,\sigma )$	1.174 ± 0.005
Logistic $\left(\nu \right)$	1.184 ± 0.005
Exponential $\left(\theta \right)$	1.227 ± 0.005
Laplace $(\mu ,\sigma )$	1.227 ± 0.005
Cauchy $(\mu ,\sigma )$	1.517 ± 0.006
Levy $(\mu ,\sigma )$	1.973 ± 0.008

, but a general proof guaranteeing $h_c\left(X\right)\ge 0$ for all distributions remains an important open question.

2.5. Multivariate and Multidimensional Generalization

We distinguish between the multivariate setting (multiple random variables $X_1,\dots ,X_n$, each potentially vector-valued) and the multidimensional setting (a single variable $\mathbf{X}\in {\mathbb{R}}^d$). For a multidimensional variable, $m\left(\mathbf{X}\right)$ is computed from kNN distances in ${\mathbb{R}}^d$. For multiple variables, each $m\left(X_i\right)$ is computed from the marginal distribution independently, and the joint measure uses the product form $m(X_1,\dots ,X_n)={\prod }_{i}m\left(X_i\right)$.

The extension to multiple variables follows naturally from the KL divergence perspective. For two random variables $(X,Y)$, the joint LDDP is:

$$H_c(X,Y)=-\int \int {\mu }_{X,Y}(x,y)\log \left({\displaystyle \frac{{\mu }_{X,Y}(x,y)}{m(x,y)}}\right)dxdy$$

(25)

Proposition 2

(Separable invariant measure). For independent scale transformations, the invariant measure for the joint distribution should satisfy:

$$m(x,y)=m_X\left(x\right)·m_Y\left(y\right)$$

(26)

where $m_X$ and $m_Y$ are the marginal invariant measures.

Theorem 2

(Joint invariant entropy). Let $(X,Y)$ be jointly distributed random variables with joint density ${\mu }_{X,Y}$, marginal densities ${\mu }_X$ and ${\mu }_Y$, and marginal invariant measures $m\left(X\right)$ and $m\left(Y\right)$ computed from the respective marginal samples. Define the normalized variables $X^{\prime }=X/m\left(X\right)$ and $Y^{\prime }=Y/m\left(Y\right)$. The invariant joint entropy is:

$$h_c(X,Y)=h\left({\displaystyle \frac{X}{m\left(X\right)}},{\displaystyle \frac{Y}{m\left(Y\right)}}\right)$$

(27)

This is invariant under independent affine transformations $(X,Y)\to (a_{1}X+b_1,a_{2}Y+b_2)$, for any $a_1,a_2\ne 0$ and $b_1,b_2\in \mathbb{R}$.

Geometric interpretation via kNN distances:

The kNN entropy estimator in 2D uses Euclidean distances. For points $A=(x_i,y_i)$ and $B=(x_j,y_j)$:

$$d(A,B)=\sqrt{{(x_i-x_j)}^2+{(y_i-y_j)}^2}$$

(28)

Under uniform transformation $(x,y)\to (kx,ky)$, distances scale as $d\to kd$. However, for independent transformations $(x,y)\to (k_{1}x,k_{2}y)$:

$$d(k_{1}A,k_{2}B)=\sqrt{k_1^2{(x_i-x_j)}^2+k_2^2{(y_i-y_j)}^2}$$

(29)

which is not simply proportional to $d(A,B)$.

By normalizing each coordinate by its invariant measure:

$$A^{\prime }=\left({\displaystyle \frac{x_i}{m\left(X\right)}},{\displaystyle \frac{y_i}{m\left(Y\right)}}\right),B^{\prime }=\left({\displaystyle \frac{x_j}{m\left(X\right)}},{\displaystyle \frac{y_j}{m\left(Y\right)}}\right)$$

(30)

The distance becomes:

$$\begin{array}{cc}\hfill d(k_1{A}^{\prime },k_2{B}^{\prime })& =\sqrt{{\left(k_1{\displaystyle \frac{x_i}{m\left(k_{1}X\right)}}-k_1{\displaystyle \frac{x_j}{m\left(k_{1}X\right)}}\right)}^2+{\left(k_2{\displaystyle \frac{y_i}{m\left(k_{2}Y\right)}}-k_2{\displaystyle \frac{y_j}{m\left(k_{2}Y\right)}}\right)}^2}\hfill \\ \hfill & =\sqrt{{\left({\displaystyle \frac{x_i-x_j}{m\left(X\right)}}\right)}^2+{\left({\displaystyle \frac{y_i-y_j}{m\left(Y\right)}}\right)}^2}\hfill \\ \hfill & =d(A^{\prime },B^{\prime })\hfill \end{array}$$

(31)

This shows that the normalized coordinates create a natural reference frame where distances are invariant under independent scale transformations—precisely what is needed for invariant entropy estimation.

By normalizing each coordinate by its invariant measure, we create a natural reference frame where distances are invariant under independent scale transformations. The invariant mutual information is then:

$$M{I}_c(X,Y)=h_c\left(X\right)+h_c\left(Y\right)-h_c(X,Y)$$

(32)

3. Results

3.1. Validation with Standard Distributions

To validate our invariant measure approach, we performed extensive simulations comparing it with traditional kNN and histogram methods. The purpose of Figure 1 is to test the core invariance property: distributions differing only in scale parameters should yield identical invariant entropy. Each column corresponds to a different distribution family (uniform, normal, exponential), with three scale parameter values overlaid. The top panels show convergence with sample size, while the bottom panels show stability across the number of neighbors k.

The key observation is that distributions with the same shape but different scale parameters yield identical invariant entropy values (overlapping curves in Figure 1). This confirms that the invariant measure successfully removes scale dependence. The convergence is rapid, with the estimation stabilizing at approximately 1500 samples. Moreover, the standard deviation is small and remains consistent across different parameter values, demonstrating the robustness of the approach.

3.2. Distribution Identification via Invariant Entropy

The median-based invariant entropy identifies distribution families independent of their location and scale parameters. Table 2 presents invariant entropy values for common continuous distributions.

Table 2 presents invariant entropy values for 14 common distribution families, ordered from lowest to highest. The ordering reflects a spectrum from highly predictable local structure (arcsine, 1.008) to highly unpredictable local structure (Lévy, 1.973). Several observations emerge. First, the invariant entropy uniquely characterizes each distribution family: all normal distributions share the same value (1.150), all exponential distributions share the same value (1.227), etc. The parameter-free nature of the invariant entropy is demonstrated in Figure 1, where curves for different scale parameters ($\sigma =[0.5,1,3]$) overlap completely. Second, the arcsine distribution has the lowest invariant entropy (1.008), even lower than the uniform distribution (1.060). This reflects the arcsine distribution’s distinctive property: its probability density concentrates at the boundaries $x=a$ and $x=b$, where ${\mu }_{\mathrm{Arcsine}}\left(x\right)=1/\left(\pi \sqrt{(x-a)(b-x)}\right)$. Given the typical nearest-neighbor spacing captured by $m\left(X\right)$, points near the boundaries are highly predictable relative to the invariant measure scale, resulting in lower entropy. Third, heavy-tailed distributions like Cauchy and Levy have higher invariant entropy, reflecting their greater unpredictability at the scale of typical nearest-neighbor distances.

3.3. Scale-Invariant Mutual Information

To demonstrate the practical advantage of invariant MI, we simulated three independent variables with vastly different scales: $X\sim \mathcal{N}(0,0.1)$, $Y\sim \mathcal{N}(0,1)$, and $Z\sim \mathcal{N}(0,10)$. These differing standard deviations highlight the advantages of the invariant entropy estimation. Since the variables are independent, the theoretical MI between any pair should be zero.

Figure 2 demonstrates striking differences between methods. The histogram method (panels a,d) achieves scale invariance, with all three MI estimates superposed in panel (a). However, convergence toward the theoretical value of zero is extremely slow as sample size increases (panel a), requiring thousands of points to approach the correct value. Furthermore, panel (d) reveals significant systematic bias that varies with the number of bins, making parameter selection critical and problematic. The bias increases substantially with larger bin counts, demonstrating a fundamental limitation of the binning approach.

The kNN method (panels b,e) shows faster convergence than histograms. In panel (b), $I(X;Y)$ and $I(Y;Z)$ are superposed and converge relatively quickly to zero. Panel (e) reveals no bias for $k<20$—the typical range preferred in the literature where smaller k values are standard. However, a severe breakdown emerges for $I(X;Z)$: in panel (e), this estimate diverges toward increasingly negative values, a physical impossibility since mutual information is non-negative by definition. This failure is not a minor numerical artifact but a fundamental problem: the divergence persists across all values of k (panel e). The breakdown occurs because the kNN estimator implicitly assumes comparable scales when computing joint nearest-neighbor distances. The 100-fold magnitude difference between X (scale ∼ 0.1) and Z (scale ∼ 10) causes the Z coordinate to completely dominate the Euclidean distance metric, corrupting the joint entropy estimation.

In contrast, the invariant method (panels c,f) demonstrates superior performance across all metrics. In panel (c), all three MI estimates ($I(X;Y)$, $I(Y;Z)$, and $I(X;Z)$) are perfectly superposed, confirming complete scale invariance regardless of the magnitude differences between variables. The convergence toward zero is faster than both the histogram and kNN methods, achieving accurate estimates with fewer than 2500 samples. Crucially, panel (f) shows no bias regardless of the number of neighbors k, eliminating the need for careful parameter tuning. The estimates remain stable and centered near zero across the entire range $k\in [3,30]$. While finite-sample estimation errors can occasionally produce small negative values near zero due to statistical fluctuations, the invariant method avoids the divergence that afflicts standard kNN estimation.

These simulations establish the superiority of the invariant approach across multiple dimensions. First, it achieves faster convergence than competing methods, requiring fewer samples to reach accurate estimates. Second, it demonstrates complete scale invariance with all MI pairs collapsing to a single curve, independent of scale differences spanning four orders of magnitude ($0.1$ to 10). Third, it exhibits no parameter-dependent bias, maintaining stability across a wide range of k values without requiring careful tuning. Fourth, it avoids the catastrophic divergence to negative values that plague traditional kNN estimation when variables have disparate scales.

3.4. Computational Efficiency

The computational complexity of our method is identical to standard kNN entropy estimation: $O\left(N\log N\right)$ for sorting and $O\left(N\right)$ for nearest neighbor search using KD-trees or ball trees. The additional computation of $m\left(X\right)$ via median is $O\left(N\right)$, making it negligible compared to the nearest neighbor search. The Julia package EntropyInvariant.jl provides an efficient implementation with performance comparable to standard kNN methods (see the Appendix A for detailed usage examples).

4. Discussion

4.1. Theoretical Contributions

This work establishes a rigorous connection between Kullback–Leibler divergence and Jaynes’s limiting density of discrete points. Our main theoretical contributions are:

• Formalization of the invariant measure: We have shown that the invariant measure $m\left(X\right)$ can be understood through the lens of KL divergence as a data-adaptive reference scale. When differential entropy is expressed relative to a uniform distribution, the implicit scale factor $\log \left(r\right)$ introduces scale-dependence. By comparing to a reference distribution with characteristic scale $m\left(X\right)$ estimated from the data itself, we remove this implicit scale dependence. This is analogous to using an empirical prior in Bayesian statistics: the reference incorporates information about the typical scale of the phenomenon.
• Bridge between discrete and continuous entropy: The KL divergence framework naturally connects Shannon’s discrete entropy to differential entropy. Our invariant measure provides the missing piece for making the continuous case behave like the discrete case with respect to invariance properties. Just as Shannon entropy for discrete variables is invariant to relabeling of outcomes, our invariant entropy for continuous variables is invariant to rescaling of measurement units.
• Rigorous change-of-variables proof: Unlike previous informal arguments, our proof of invariance (Theorem 1) explicitly uses the change-of-variables formula for probability densities. This establishes the result on solid mathematical foundations and clarifies the role of the Jacobian in transformation properties.
• Multivariate generalization with geometric interpretation: The extension to joint distributions follows naturally from the separability principle in KL divergence (Proposition 2). The geometric interpretation shows that normalizing by marginal invariant measures creates a coordinate system where Euclidean distances are invariant under independent scale transformations, a property essential for multivariate kNN entropy estimation.

4.2. Comparison with Existing Approaches

Traditional histogram methods: As demonstrated in the original work and Figure 2, histogram methods suffer from systematic bias that increases with the number of bins. There is no principled way to choose the optimal bin width, and the method shows severe scale sensitivity. Our approach eliminates these issues by using a data-adaptive scale.

Standard kNN estimator: The standard kNN estimator [3] provides consistent entropy estimates and is computationally efficient. However, it is not scale-invariant, as evidenced by the negative MI values in Figure 2d. Our approach modifies this by normalizing data by $m\left(X\right)$ before applying the kNN estimator, yielding ${\widehat{h}}_c\left(X\right)=\widehat{h}(X/m\left(X\right))$. This simple modification preserves all the advantages of kNN methods while adding scale invariance.

Kernel density estimators: KDE methods [8] face similar bandwidth selection challenges as histogram methods. While sophisticated adaptive bandwidth selection procedures exist, they add computational complexity. Our median-based invariant measure provides a simple, robust alternative that requires no tuning beyond the standard k parameter.

Nagel et al.’s approach: Nagel and coworkers [15] proposed normalizing MI by subtracting a scaling factor computed from marginal entropies. However, their normalization has a fundamental flaw: it affects marginal entropies differently depending on which variables are paired together. For example, the normalized entropy of X when computed with Y differs from its normalized entropy when computed with Z. This violates the principle that a variable’s entropy should be an intrinsic property. Our approach provides consistent entropy for each variable regardless of which other variables it is paired with, because each variable has its own invariant measure $m\left(X\right)$ computed from its marginal distribution.

4.3. Interpretation of Results

Our simulation results demonstrate several key properties:

1.

Fast convergence with small samples: The invariant estimator converges faster than traditional methods, particularly when variables have different scales (Figure 2, panels e,f versus c,d). This occurs because normalization by $m\left(X\right)$ and $m\left(Y\right)$ brings variables to comparable scales before computing joint entropy. The kNN distance calculations then operate in a balanced space where all dimensions contribute equally.

2.

Distribution identification: Table 2 shows that distributions maintain consistent invariant entropy values regardless of location and scale parameters. From the KL divergence perspective, this makes sense: distributions in the same family (e.g., all normal distributions) differ only in location parameter $\mu$ and scale parameter $\sigma$. The invariant measure removes precisely these parameters, leaving only the intrinsic “shape” of the distribution. This property enables distribution classification based solely on shape characteristics.

3.

Boundary concentration in arcsine distribution: The arcsine distribution has the lowest invariant entropy (1.008), even lower than uniform (1.060). This initially surprising result reflects a deep property of the arcsine distribution. Its density $\mu \left(x\right)=1/\left(\pi \sqrt{(x-a)(b-x)}\right)$ diverges at the boundaries, meaning probability mass concentrates there. When we measure entropy relative to the typical nearest-neighbor spacing $m\left(X\right)$, points near boundaries are highly predictable, and their neighbors must also be near the boundary. This local predictability, captured by the invariant measure, results in lower entropy despite the distribution appearing “spread out” over $[a,b]$.

4.4. Relationship to Maximum Entropy Principle

The Maximum Entropy (MaxEnt) principle and our invariant entropy framework address different questions. MaxEnt is a distribution selection principle: given constraints (e.g., fixed mean, fixed variance), it selects the distribution that maximizes entropy, yielding the “least biased” distribution compatible with the constraints. Invariant entropy is an entropy measurement principle: given a distribution (known or empirically observed), it computes an entropy value that is invariant to measurement units. These frameworks are complementary rather than competing.

The MaxEnt principle yields different distributions depending on the constraint structure:

Bounded support constraint: Among all distributions with support contained in a fixed interval $[a,b]$, the uniform distribution $\mathcal{U}(a,b)$ maximizes differential entropy:

$$h_{\mathrm{uniform}}=\log (b-a)$$

For the uniform distribution, the variance ${\sigma }^2={(b-a)}^2/12$ is determined by the interval bounds, not independently specified.

Fixed variance constraint on $\mathbb{R}$: Among all distributions on $\mathbb{R}$ with fixed variance ${\sigma }^2$, the normal distribution $\mathcal{N}(\mu ,\sigma )$ maximizes differential entropy:

$$h_{\mathrm{normal}}={\displaystyle \frac{1}{2}}\log \left(2\pi e{\sigma }^2\right)$$

This is fundamentally different: the domain is unbounded, and variance is an independent constraint.

From the invariant entropy perspective, all uniform distributions yield $h_c\approx 1.060$ and all normal distributions yield $h_c\approx 1.150$, regardless of their parameters (Table 2). The uniform has relatively low invariant entropy (1.060). This reflects what the invariant measure captures: the median nearest-neighbor distance reflects the typical local spacing of points. For a uniform distribution, this spacing is highly regular—when we normalize by $m\left(X\right)$, all points lie within a predictable range relative to their typical spacing. In contrast, heavy-tailed distributions like Cauchy and Levy exhibit extreme variability in local density: some regions have tightly clustered points while others are sparse. This variability persists after normalization, yielding higher invariant entropy. Distributions with high standard differential entropy (given appropriate constraints) also tend to have high invariant entropy, suggesting consistency between the two frameworks rather than competition.

4.5. Connections to Information Geometry

From the perspective of information geometry [16], our approach can be understood as choosing a coordinate system on the manifold of probability distributions. The Fisher information metric provides a natural Riemannian structure on this manifold, and geodesics correspond to exponential families.

The invariant measure $m\left(X\right)$ defines a natural coordinate chart that makes entropy calculations coordinate-independent, analogous to working in canonical coordinates in differential geometry. For a family of distributions $\{p_{\theta }:\theta \in \mathsf{\Theta }\}$, the Fisher metric is:

$$g_{ij}\left(\theta \right)=\mathbb{E}\left[{\displaystyle \frac{\partial \log p_{\theta }}{\partial {\theta }_i}}{\displaystyle \frac{\partial \log p_{\theta }}{\partial {\theta }_j}}\right]$$

(33)

For location-scale families $p_{\mu ,\sigma }\left(x\right)={\displaystyle \frac{1}{\sigma }}{p}_0\left({\displaystyle \frac{x-\mu }{\sigma }}\right)$, the invariant measure removes the $(\mu ,\sigma )$ dependence, effectively projecting onto the “shape manifold” of distributions. Future work could explore whether there is a canonical connection between our invariant measure and the Fisher–Rao metric, potentially leading to a geometric interpretation of the LDDP as arc length on the shape manifold.

4.6. Limitations and Future Directions

1. Beyond affine transformations: Our current framework handles affine transformations $(x,y)\to (ax+b,cy+d)$. Extending to general diffeomorphisms $\varphi :{\mathbb{R}}^d\to {\mathbb{R}}^d$ would require incorporating the Jacobian determinant $\left|\det D\varphi \right|$. The invariant measure would need to transform as $m\left(\varphi \left(X\right)\right)=m\left(X\right)·{\left|\det D\varphi \right|}^{1/d}$ to preserve invariance. This generalization could connect to the theory of differential forms and volume elements in differential geometry.

2. Theoretical optimality of the median: While our empirical results demonstrate that the median of nearest-neighbor distances avoids negative entropy and satisfies all required invariance properties, a complete mathematical characterization remains to be established. The median is optimal as an $L^1$ center (minimizing $\sum |d_i-c|$), while the mean is optimal as an $L^2$ center (minimizing $\sum {(d_i-c)}^2$). Different $L^p$ centers induce different geometries on the distance distribution, each preserving the geometric invariance properties but potentially yielding different entropy values. Establishing the minimax optimality of the median among all $L^p$ centers for minimizing the probability of negative entropy across relevant distribution classes, and characterizing the tail conditions under which $P(h_c<0)\to 0$ as $N\to \infty$, would provide a rigorous theoretical foundation. Such an analysis could parallel the development of M-estimators in robust statistics, potentially establishing conditions under which the $L^1$ geometry provides the natural reference scale for entropy measurement.

3. Connection to rate-distortion theory: Rate-distortion theory [12] involves minimizing $I(X;Y)$ subject to distortion constraints $\mathbb{E}\left[d\right(X,Y\left)\right]\le D$. The invariant MI might provide new insights into scale-invariant coding schemes where the distortion metric itself adapts to the data scale. This could have applications in lossy compression where preservation of “shape” is more important than absolute accuracy.

4. Applications in causality: Invariance under interventions is central to causal inference. Pearl’s do-calculus and the invariance principle of Peters et al. both rely on identifying relationships that remain stable across environments. Our scale-invariant MI might help identify causal relationships that persist across different measurement scales or units, potentially improving causal discovery algorithms when variables are measured inconsistently across datasets.

5. Extensions to discrete–continuous mixtures: Many real-world datasets contain both discrete and continuous variables. The MI between discrete and continuous variables is well-defined, but estimating it is challenging [2]. The invariant measure framework could potentially be extended to mixed data types by defining appropriate reference measures for discrete components.

5. Conclusions

This work establishes a rigorous theoretical foundation for invariant entropy estimation by connecting Jaynes’s limiting density of discrete points to the Kullback–Leibler divergence framework. By defining the invariant measure $m\left(X\right)$ as the median of nearest-neighbor distances and proving that $h_c\left(X\right)=h(X/m\left(X\right))$ is truly scale-invariant, we provide the first practical method for computing Jaynes’s LDDP. The approach extends naturally to multivariate settings and demonstrates superior performance compared to standard methods, particularly avoiding catastrophic failures when variables have disparate scales.

The invariant entropy represents a logical evolution of Shannon’s information theory for continuous variables. Just as Shannon entropy for discrete variables is invariant to relabeling of outcomes, our invariant entropy for continuous variables is invariant to rescaling of measurement units. This makes it a more natural measure of uncertainty for physical quantities that can be measured in different units—temperature in Celsius versus Fahrenheit, distance in meters versus feet, concentration in molarity versus parts-per-million.

Beyond practical utility, the invariant entropy offers theoretical insights. Table 2 reveals that the arcsine distribution, despite appearing spread over an interval, has the lowest invariant entropy due to boundary concentration effects visible at the nearest-neighbor scale. Conversely, heavy-tailed distributions like Cauchy and Levy have high invariant entropy, reflecting fundamental unpredictability even at local scales. These observations suggest that invariant entropy captures the intrinsic properties of distribution families independent of parametrization.

By grounding this concept in the well-established KL divergence framework, we provide both theoretical justification and practical tools for scale-invariant information-theoretic analysis. The connection to information geometry suggests deeper links between invariant measures and the Fisher–Rao metric on distribution manifolds, opening avenues for future research.

We hope this work will enable new applications across diverse scientific fields where scale invariance is essential: feature selection in machine learning with mixed-unit data, network inference in systems biology where genes have vastly different expression scales, time series analysis comparing signals with different amplitudes, and causal discovery across heterogeneous datasets. Open-source implementations in Julia (EntropyInvariant.jl) and Python (entropy_invariant) make these methods readily accessible to the research community.