Kolmogorov GAM Networks Are All You Need!

Abstract

Kolmogorov GAM (K-GAM) networks have been shown to be an efficient architecture for both training and inference. They are additive models with embeddings that are independent of the target function of interest. They provide an alternative to Transformer architectures. They are the machine learning version of Kolmogorov’s superposition theorem (KST), which provides an efficient representation of multivariate functions. Such representations are useful in machine learning for encoding dictionaries (a.k.a. “look-up” tables). KST theory also provides a representation based on translates of the Köppen function. The goal of our paper is to interpret this representation in a machine learning context for applications in artificial intelligence (AI). Our architecture is equivalent to a topological embedding, which is independent of the function, together with an additive layer that uses a generalized additive model (GAM). This provides a class of learning procedures with far fewer parameters than current deep learning algorithms. Implementation can be parallelizable, which makes our algorithms computationally attractive. To illustrate our methodology, we use the iris data from statistical learning. We also show that our additive model with non-linear embedding provides an alternative to Transformer architectures, which, from a statistical viewpoint, are kernel smoothers. Additive KAN models, therefore, provide a natural alternative to Transformers. Finally, we conclude with directions for future research.

1. Introduction

The landscape of modern machine learning has been shaped by exponential growth in computational power, particularly through advances in GPU technology and frameworks like PyTorch. While Moore’s law continues to drive hardware improvements, and CUDA algorithms have revolutionized our ability to process vast amounts of internet data, we pose the following question: can we achieve superior performance through mathematically efficient representations of multivariate functions rather than raw computational power?

In this paper, we introduce Kolmogorov-generalized additive models (K-GAM), a novel neural network architecture whose additive structure enables simplified training and accelerated inference through a marked reduction in parameters compared to traditional approaches. K-GAMs leverage Kolmogorov’s superposition theorem by utilizing a composition of two key components, namely, a universal Köppen embedding (functioning as a space-filling curve), followed by a trainable outer function, $g:[0,1]\to [0,1]$. Unlike the original iterative look-up table approach proposed by Köppen and Sprecher, which faces NP-hard computational challenges, we implement the outer function using a ReLU neural network that can be efficiently trained using standard optimization techniques.

A fundamental challenge in machine learning lies in effectively handling high-dimensional input–output relationships. This challenge manifests itself in two distinct but related tasks. First, one task is to construct a “look-up” table (dictionary) for the fast search and retrieval of input–output examples. This is an encoding and can be thought of as a data compression problem. Second, and perhaps more importantly, we must develop prediction rules that can generalize beyond these examples to handle arbitrary inputs.

More formally, we seek to find a good predictor function $f\left(x\right)$ that maps an input x to its output prediction y. In practice, the input x is typically a high-dimensional vector:

$$y=f\left(x\right)\mathrm{where}x=(x_1,\dots ,x_d)$$

Given a training dataset ${(y_i,x_i)}_{i=1}^N$ of example input–output pairs, our goal is to train a model, i.e., to find the function f. The key question is as follows: how do we represent a multivariate function so as to obtain a desirable f?

We demonstrate that Transformer architectures can be decomposed into two fundamental operations: a particular form of embedding followed by kernel smoothing. Hence, K-GAM models are natural competitors to Transformers. The significance of this connection becomes clear when we consider that Transformers have largely supplanted alternative architectures across a wide range of applications. Previous work in this direction—[1,2]—demonstrated the potential of Köppen-based approaches only for $2\times 2$ image reconstruction tasks. Our analysis extends this framework to encompass A wider spectrum of machine learning problems.

The rest of this paper is outlined as follows. Section 2 discusses the Kolmogorov superposition theorem. Section 3 introduces Kolmogorov-generalized additive models (K-GAMs). Section 4 describes Transformer architectures as kernel smoothers. Section 5 provides an application of GAM-Kolmogorov embeddings to the iris dataset. Section 6 concludes with directions for future research.

2. Kolmogorov Superposition Theorem (KST)

Kolmogorov demonstrated that any real-valued continuous function $f\left(\mathbf{x}\right)$ where $\mathbf{x}=(x_1,\dots ,x_d)$ defined on $E^d$ can be represented as a convolution of two single-variable functions:

$$f(x_1,\dots ,x_d)=\sum _{q=0}^{2d}{g}_q\left({\Psi }_q(x_1,\dots ,x_d)\right)$$

(1)

where $g_q$ are continuous single-variable functions defined on the d-dimensional unit cube $E^d={[0,1]}^d$. Kolmogorov further showed that the ${\varphi }_q$ functions can be decomposed into sums of single-variable functions:

$${\Psi }_q(x_1,\dots ,x_d)=\sum _{i=1}^d{\psi }_{q,i}\left(x_i\right)$$

(2)

This result is known as the Kolmogorov representation theorem [3] and is often written in the following form:

$$f(x_1,\dots ,x_d)=\sum _{q=0}^{2d}{g}_q\left(\sum _{i=1}^d{\psi }_{q,i}\left(x_i\right)\right)$$

(3)

Kolmogorov originally defined the inner functions as ${\Psi }_q(x_1,\dots ,x_d)$, $q=0,1,\dots ,2d$, using the following property ${\psi }_q\left(S_{q,k}\right)\cap {\psi }_{q^{\prime }}\left(S_{q,k^{\prime }}\right)=\varnothing ,k\ne k^{\prime },q,=0,1,\dots ,2d$. Then, for any integer $d\ge 2$, let $S_{q,k}$ be a family of mutually disjoint d-dimensional cubes $S_{q,k}\cap S_{q,k^{\prime }}=\varnothing ,k\ne k^{\prime },k,k^{\prime }=1,2,\dots ,q=0,1,\dots ,2d$. The diameter of each cube approaches zero as $k\to \infty$, and every point of a unit cube ${[0,1]}^d$ belongs to at least $d+1$ cubes $S_{q,k}$ for each k. Unfortunately, this derivation is not constructive and does not provide a practical algorithm for constructing the inner functions ${\varphi }_q$.

Our goal is to show that it applies directly to the problems of machine learning. Furthermore, we interpret the architecture as a statistical model that allows us to draw on uncertainty quantification and approximation bounds. To understand the power of this representation from a machine learning perspective, we can think of ${\psi }_{q,i}\left(x_i\right)$ as features. Note that the number of features is then $2d+1$ for the d-dimensional input. This representation is also remarkable in the sense that ${\psi }_q\left(x\right)$ can be viewed as an embedding, which is independent of f. For details, see Section 3.

2.1. Kolmogorov–Arnold Networks

A significant development has been the emergence of Kolmogorov–Arnold networks (KANs). The key innovation of KANs is their use of learnable functions rather than weights on the network edges. This replaces traditional linear weights with univariate functions, typically parametrized by splines, enhancing both representational capacity and interpretability. Ref. [4] established the first practical connection between KST and neural networks by showing that any KAN can be constructed as a three-layer MLP. Polson and Sokolov proposed Bayesian KAN networks.

Recently, Refs. [5,6] considered KST in the form of a sum of functions, represented as a two-layer model:

$$f\left(\mathbf{x}\right)=f(x_1,\dots ,x_d)=(g\circ \Phi )\left(x\right)$$

(4)

More explicitly,

$$f(x_1,\dots ,x_d)=\sum _{q=0}^{2d}\sum _{p=1}^d{\psi }_{q,p}\left(x_p\right)$$

(5)

Ref. [7] presented a significant theorem regarding the approximation of generalized bandlimited multivariate functions using deep ReLU networks. The key finding is that deep ReLU networks can approximate generalized bandlimited functions without suffering from the curse of dimensionality. The paper utilizes an extension of Carathéodory’s theorem to infinite-dimensional spaces, which is crucial for the approximation of bandlimited functions. As a result, this paper provides a theoretical foundation for the effectiveness of deep learning in high-dimensional problems, especially for a class of functions relevant to many scientific computing applications.

One can think of KAN networks as NNs in which functions ${\psi }_{q,p}$ are estimated at the nodes, rather than weights. This makes them very flexible. A natural choice involves the use of splines for ${\psi }_{q,p}$; Ref. [8] proposed the application of KSNs (Kolmogorov spline networks).

Another advantage is that they can be trained using fast, high-dimensional algorithms such as the Newton–Kaczmarz algorithm [9]. The authors showed that KAN is not only a superposition of functions but also a special case of a tree of discrete Urysohn operators:

$$U(x_1,\dots ,x_d)=\sum _{p=1}^d{g}_p\left(x_p\right)$$

(6)

This insight leads to a fast, scalable algorithm that avoids backpropagation, is applicable to any GAM model, and uses a projection descent method with a Newton–Kaczmarz scheme.

Recent research has explored various functional classes for the outer functions, including Wan-KAN [10], SineKAN [11], and functional combinations [12]. Ref. [13] demonstrated that KAN networks can outperform traditional MLPs, while [14] proved their optimality in terms of approximation errors. Ref. [15] discussed the Bayesian interpretation of the Kolmogorov–Arnold representation. KAN networks are widely trained with gradient descent. Ref. [9] proposed a Newton–Kaczmarz scheme. Bayesian methods are also available [16] and offer the potential for fewer parameters and more efficient inference. There are also optimal posterior concentration results [17].

Early theoretical groundwork for adaptive learning in KANs was laid by [18], who introduced the concept of counter-propagation networks, where outer functions self-organize in response to input–output pairs $(x,y)$. Theoretical understanding has continued to advance, with [19] showing that Kolmogorov networks with two hidden layers can precisely represent continuous, discontinuous, and unbounded multivariate functions, depending on the activation function choice in the second layer. Ref. [20] extended this to three-layer networks for discontinuous functions.

2.2. Inner and Outer Functions

The theorem has seen several refinements over time. Ref. [21] made a crucial advancement by proving that the inner functions could be Hölder continuous. Ref. [22] later strengthened this result, showing that these functions could be Lipschitz continuous, although this required modifications to both the outer and inner functions.

We focus on the Kolmogorov–Sprecher formulation, as modified by [23,24,25]. The first representation is as follows:

$$f(x_1,\dots ,x_d)=\sum _{q=0}^{2d}{g}_q\left({\Psi }_q\left(x\right)\right)\mathrm{where}{\Psi }_q\left(x\right)=\sum _{p=1}^d{\alpha }_p\psi (x_p+q\alpha )$$

(7)

We will call this the Kq-GAM model to indicate the fact that there are q univariate outer functions. Ref. [7] provided an iterative algorithm to find $g_q$ based on the original Köppen formulation.

The second (and most parsimonious) representation is one in which there is only one univariate outer function! This expresses any multivariate continuous function as follows:

$$f(x_1,\dots ,x_d)=\sum _{q=0}^{2d}g\left({\Psi }_q\left(x\right)\right)\mathrm{where}{\Psi }_q\left(x\right)=\sum _{p=1}^d{\alpha }_p\psi (x_p+qa)+{\delta }_q.$$

(8)

Ref. [25] defines the ${\delta }_k$ term.

We will refer to this as the K-GAM model in our ML implementation. Bryant discussed the choice of offset ${\delta }_q$. Note that the complexity reduction is that there is only one $g:[0,1]\to [0,1]$ outer function that depends on f. The caveat involves finding a computation algorithm to compute g in polynomial time.

Here, the inner functions ${\psi }_q$ are constructed as sums of translated versions of a single univariate function $\psi$, known as the Köppen function. The Köppen function is monotone, Hölder continuous with smoothness parameters, and has fractal-like properties. Recent work by [26] provides a modern construction of the Köppen function and establishes theoretical approximation results with ReLU networks. Ref. [27] extended these theoretical results and showed that a function with a smaller number of parameters can be used. The author used $\lambda$-adic representation, where

$$f(x_1,\dots ,x_d)=g\left(\sum _{p=1}^d{\lambda }^{-p}\psi \left(x_p\right)\right).$$

(9)

The structure of ${\psi }_q$ differs fundamentally from the approaches used in traditional neural networks, which rely on hyperplanes (as in ReLU networks) or decision trees. In ${\psi }_q$, a slice in any given univariate direction cuts the regions defined by the embedding in half.

The inner functions, ${\psi }_q$, partition the input space into distinct regions, and the outer function, g, must be constructed to provide the correct output values across the regions that the inner function defines. The outer function, g, can be determined via a computationally intensive process of averaging. For each input configuration, the inner functions, ${\psi }_q$, generate a unique encoding, and g must map this encoding to the appropriate value of $f\left(x\right)$. This creates a dictionary-like structure that associates each region with its corresponding output value. Köppen made significant contributions by correcting Sprecher’s original proof of this construction process, with improvements to the computational algorithm later suggested by [28,29]. Braun further enhanced the understanding by providing precise definitions of the shift parameters ${\delta }_k$ and characterizing the topological structure induced by ${\psi }_q$.

A fundamental trade-off in KST exists between function smoothness and dimensionality. The inner functions, ${\psi }_{p,q}$, can be chosen from two different function spaces, with each offering distinct advantages. The first option is to use functions from $C^1\left([0,1]\right)$, but this limits the network’s ability to handle higher dimensions effectively. The second option is to relax the smoothness requirement to Hölder-continuous functions (${\psi }_{p,q}\in {\mathrm{Holder}}_{\beta }\left([0,1]\right)$), which satisfy the inequality ${|\psi \left(x\right)-\psi \left(y\right)|<|x-y|}^{\beta }$. These functions are less smooth, but this “roughness” enables better approximation in higher dimensions.

2.3. Ridge and Projection Pursuit Regression

To understand the significance of this trade-off, we consider ridge functions, which represent a fundamental building block in multivariate analysis. Since our ultimate goal is to model arbitrary multivariate functions f, we need a way to reduce dimensionality while preserving the ability to capture nonlinear relationships. Ridge functions accomplish this by representing one of the simplest forms of nonlinear multivariate functions, requiring only a single linear projection and a univariate nonlinear transformation. Formally, a ridge function $f:{\mathbb{R}}^d\to \mathbb{R}$ takes the form $f\left(x\right)=g\left(w^{T}x\right)$, where g is a univariate function and $x,w\in {\mathbb{R}}^d$. The non-zero vector w is called the direction. The term “ridge” reflects a key geometric property, that is, the function remains constant along any direction orthogonal to w. Specifically, for any direction u, such that $w^{T}u=0$, we have the following:

$$f(x+u)=g\left(w^T(x+u)\right)=g\left(w^{T}x\right)=f\left(x\right)$$

(10)

This structural simplicity makes ridge functions particularly useful as building blocks for high-dimensional approximation. Ref. [30] provides bounds on the approximation error when a function with a finite spectral norm is approximated by ridge functions.

Ridge functions play a central role in high-dimensional statistical analysis. For example, projection pursuit regression approximates input–output relations using a linear combination of ridge functions [31,32,33]:

$$f\left(x\right)=\sum _{q=0}^{2d}{g}_q\left(w_q^{T}x\right),$$

(11)

where both the directions, $w_i$, and functions, $g_i$, are variables, and $w_i^{T}x$ denotes one-dimensional projections of the input vector. The vector $w_i^{T}x$ is a projection of the input vector x onto a one-dimensional space, and $g_i\left(w_i^{T}x\right)$ denotes a feature calculated from the data. Ref. [34] used nonlinear functions of linear combinations, laying important groundwork for deep learning.

3. Kolmogorov-Generalized Additive Models (K-GAMs)

We take a different approach. Rather than using learnable functions as network node activations, we directly use the KST representation. This is a two-layer network with a non-differentiable inner function. The network’s architecture can be expressed as follows:

$$f(x_1,\dots ,x_d)=\sum _{q=0}^{2d}{g}_q\left(z_q\right)$$

(12)

where the inner layer performs an embedding from ${[0,1]}^d$ to ${\mathbb{R}}^{2d+1}$ via the following;

$$z_q={\eta }_q(x_1,\dots ,x_d)=\sum _{p=1}^d{\lambda }_p\psi (x_p+qa)$$

(13)

Here, ${\lambda }_p={\sum }_{r=1}^{\infty }{\gamma }^{-(p-1)\beta \left(r\right)}$ is a p-adic expansion with $\beta \left(r\right)=(d^r-1)/(d-1)$ and $\gamma \ge d+2$, with $a={\left(\gamma (\gamma -1)\right)}^{-1}$.

3.1. Köppen Function

The Köppen function $\psi$ straightforwardly computes and provides the basis for the inner architecture of the network. The function $\psi$ is defined through a recursive limit:

$$\psi \left(x\right)=\underset{k\to \infty }{lim}{\psi }_k\left(\sum _{l=1}^k{i}_l{\gamma }^{-l}\right)$$

(14)

where each $x\in [0,1]$ has the following representation:

$$x=\sum _{l=1}^{\infty }{i}_l{\gamma }^{-l}=\underset{k\to \infty }{lim}\left(\sum _{l=1}^k{i}_l{\gamma }^{-l}\right)$$

(15)

and ${\psi }_k$ is defined recursively as follows:

$${\psi }_k\left(d\right)=\left\{\begin{array}{cc}d,\hfill & d\in D_1\hfill \\ {\psi }_{k-1}(d-i_k{\gamma }^{-k})+i_k{\gamma }^{-{\beta }_d\left(k\right)},\hfill & d\in D_k,k>1,i_k<\gamma -1\hfill \\ {\displaystyle \frac{1}{2}}\left({\psi }_{k-1}(d-i_k{\gamma }^{-k})+{\psi }_{k-1}(d+{\gamma }^{-k})\right)+i_k{\gamma }^{-{\beta }_d\left(k\right)},\hfill & d\in D_k,k>1,i_k=\gamma -1\hfill \end{array}\right.$$

(16)

Here, the sets $D_k$ are defined as follows. $D_k$ is a uniform grid, with step size ${\gamma }^{-k}$, defined by the following:

$$D+k=\{i{\gamma }^{-k},0\le i\le {\gamma }^k-1\}\subset [0,1)$$

(17)

There are ${\gamma }^k$ different points $0\le x\le 1-{\gamma }^{-k}$ on each grid $D_k$ where each point is as follows: $x={\sum }_{l=1}^l{i}_l{\gamma }^{-l},i\in \{0,1,\dots ,\gamma -1\}$.

To illustrate, we show how the iterates update. Figure 1 shows the plot of the Köppen function ${\psi }_k$ for $k=3$ on intervals $[0,1]$ and $[0,0.2]$ (top row). The bottom row shows a “zoomed in” view of the function for $k=4$ and $k=5$ on the intervals $[0,0.2]$. The function has a fractal-like structure, with the number of discontinuities increasing with k. The Köppen function is a key component of the KST representation, providing a topological embedding of the input space that is independent of the target function f. This embedding is crucial for the network’s ability to capture complex multivariate relationships. Ref. [28] provided a detailed algorithm for computing the Köppen function and its properties. The author also provided a good, intuitive understanding of the function and its properties.

The most striking aspect of KST is that it leads to a generalized additive model (GAM) with fixed features that are independent of the target function f. These features, determined by the Köppen function, provide universal topological information about the input space, effectively implementing a k-nearest neighbors structure that is inherent to the representation. The outer function, g, is then responsible for learning the relationship between these features and the target function f. This separation of feature engineering and learning is a key advantage of K-GAM networks, enabling efficient training and inference.

Theorem 1

(K-GAM). Any function and dataset can be represented as a GAM with feature engineering (topological information) given by features $z_k$ in the hidden layer:

$$\begin{array}{cc}\hfill y^{\left(i\right)}& =\sum _{q=0}^{2d}g\left(z_q^{\left(i\right)}\right)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill z_q^{\left(i\right)}& =\sum _{p=1}^d{\lambda }_q\psi (x_p^{\left(i\right)}+qa)+{\delta }_q\hfill \end{array}$$

(19)

where ψ is a single activation function common to all nodes, known as the Köppen function, and g is a single outer function.

Proof. 

We can take the Köppen–Sprecher representation for $y=f\left(x\right)$ and apply it N times to each output–input pair $(y^{\left(i\right)},x^{\left(i\right)})$ and obtain a representation for $y^{\left(i\right)}=f\left(x^{\left(i\right)}\right)$. For example, for the K-GAM model and g-outer function, we have the following:

$$f(x_1,\dots ,x_d)=\sum _{q=0}^{2d}g\left(\sum _{p=1}^d{\lambda }_q\psi (x_p+qa)+{\delta }_q\right)$$

where ${\lambda }_1=1>{\lambda }_2\dots >{\lambda }_d$ and $a=1/(2d+1)(2d+2)$.

Now, for a dataset ${\left\{(y^{\left(i\right)},x^{\left(i\right)})\right\}}_{i=1}^N$ where $y^{\left(i\right)}=f\left(x^{\left(i\right)}\right)$, we apply this representation to each data point:

$$\begin{array}{cc}\hfill y^{\left(i\right)}& =f(x_1^{\left(i\right)},\dots ,x_d^{\left(i\right)})\hfill \\ \hfill & =\sum _{q=0}^{2d}g\left(\sum _{p=1}^d{\lambda }^q\varphi (x_p^{\left(i\right)}+pa)+{\delta }_q\right)\hfill \\ \hfill & =\sum _{q=0}^{2d}g\left(z_q^{\left(i\right)}\right)\hfill \end{array}$$

where we define $z_q^{\left(i\right)}={\sum }_{p=1}^d{\lambda }_q\varphi (x_p^{\left(i\right)}+pa)+{\delta }_q$.

Ref. [7] proved that the outer function g can be approximated by a ReLU network to arbitrary precision. Therefore, we can replace g with a single ReLU network:

$$g\left(x\right)=\sum _{k=1}^K{\beta }_k\mathrm{ReLU}(w_{k}x+b_k),$$

where K is the number of neurons. We see that any function, f, and its corresponding dataset can be represented as a GAM with features engineered through the Köppen function $\varphi$.    □

Ref. [35] further developed this insight into a specialized two-layer ReLU network architecture. Specifically, the authors introduced a class of functions called Kolmogorov–Lipschitz continuous, where the outer function in the KST representation is Lipschitz continuous. Then they demonstrated that KL continuous functions can be approximated using a ReLU neural network with two hidden layers, achieving a dimension-independent approximation rate, and they introduced LKB-splines, created by using linear B-splines to replace the outer function in the KST representation.

As in [36], we can decompose with Lipschitz continuous functions. Note, the number of outer functions grows with the input dimension, d, not with the dataset size. The outer function, g, is shared across all terms, making this a true GAM representation with universal feature engineering provided by the inner Köppen function $\varphi$.

In a similar vein to [4], we show how this functional representation can be used to define a statistical model for applications in machine learning tasks [37,38]. We propose the following model:

$$\begin{array}{cc}\hfill f(x_1,\dots ,x_d)& =\sum _{q=0}^{2d}g\left(\sum _{p=1}^d{\alpha }_p\psi (x_p+q\alpha )+{\delta }_q\right)\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill g\left(x\right)& =\sum _{k=1}^K{\beta }_k\mathrm{ReLU}(w_{k}x+b_k)\hfill \end{array}$$

(21)

where K denotes the number of neurons in the outer function’s architecture. The parameters ${({\beta }_k,w_k,b_k)}_{k=1}^K$ can be learned using $L^2$-minimization and SGD. This architecture employs only two univariate activation functions, that is, a learned ReLU network for the outer layer and the Köppen function for the inner layer, which can be computed independently of f.

Note

A crucial aspect of KST-based architectures is their relationship to p-adic expansions and embeddings. This connection, first explored by [39] and recently advanced by [40], builds upon the foundational work of [41] on cellular neural networks in the 1980s, which included cellular automata as special cases.

For any input x, its p-adic expansion can be written as follows:

$$x=\sum _{l=1}^{\infty }{i}_l{\gamma }^{-l}=\underset{k\to \infty }{lim}\left(\sum _{l=1}^k{i}_l{\gamma }^{-l}\right)$$

(22)

where $\gamma$ serves as the base of the expansion and $i_l$ denotes the digits in this representation. This expansion provides a natural embedding of the input space into a higher-dimensional representation, resembling contemporary kernel embedding techniques in deep learning [15,37,38].

3.2. Inference

An important property of an algorithm is how many parameters exist in the evaluation of the network, the so-called inference problem. The K-GAM network is notable for its relatively sparse parameterization.

Consider the class that corresponds to kernel estimators of the following form:

$$f\left(x\right)=\int K(x,t)d\mu \left(t\right)\mathrm{with}\int d\mu \left(t\right)\le K\mathrm{and}x=(x_1,\dots ,x_d)$$

We call $K(x,t)=K_t\left(x\right)$ an $L_{\infty }$-atom if $|K_t\left(x\right)|\le 1$. For such estimators, there exists an M-term approximation of f:

$$f_M\left(x\right)=\sum _{j=1}^M{a}_{j}K(x,t_j)\mathrm{such}\mathrm{that}{\left|\left|f-f_N\right|\right|}_{\infty }\le C{M}^{-{\displaystyle \frac{1}{2}}},$$

with C independent of dimension d. While general approximations suffer from the curse of dimensionality with error $O\left(M^{-r/d}\right)$ [42], where r denotes isotropic smoothness, certain restrictions yield dimension-independent bounds. For instance, when $\nabla f$ is in the $L^1$-Fourier class, the error becomes $O\left(M^{-1/2}\right)$ independent of d [43].

The class of functions that we consider can be represented as $L^1$-combinations of $L^{\infty }$-atoms. For $f_M$, we will use superpositions of functions. For example, with $t:=(x_0,s)$ and Gaussian atoms $K(x,t)=exp\left(-{\left|\left|x-x_0\right|\right|}^2/s^2\right)$, see [44]. Let $t=(x_0,k)$, where k is an orthant indicator. Then the resultant approximation is also of order $O\left(M^{-1/2}\right)$, independent of d.

This framework naturally extends to incorporate dimension-reduction techniques like partial least squares (PLS). Given the input–output pairs ${(x_i,y_i)}_{i=1}^N$, we can construct models of the following form:

$$Y=g\left({\widehat{B}}_{PLS}X\right)$$

where ${\widehat{B}}_{PLS}$ provides dimension reduction and g is a learnable transformation.

A more general class is found by the superposition of $2^d$ functions, each monotone for a different orthant. Then, the condition $\int d\left|\mu \right|\left(t\right)\le 1$ manifests as a derivative condition:

$${\displaystyle \frac{{\partial }^{d}f}{\partial x_1\dots \partial x_d}}\in L_1.$$

This characterizes the space of functions with bounded mixed derivatives, which naturally leads to $\beta$-Hölder-smooth functions. For such $\beta$-Hölder-continuous functions, the network typically requires $O\left(M^{\beta }\right)$ parameters (see [45] for further results on additive models and approximation properties in high dimensions).

Kolmogorov spline networks, designed for differentiable functions with bounded derivatives, satisfy the stronger property:

$$\left|\left|f-f_M\right|\right|=O\left(M^{-1}\right)$$

with the number of parameters given by $O\left(M^{2/3}\right)$. This compares favorably to both the $O\left(M^{-1/2}\right)$ approximation rate and the $O\left(M^2\right)$ parameter count of the general one-layer-hidden feed-forward networks [8].

Deep ReLU networks, using $max(x,0)$ activation, create separating hyperplanes across L layers, unlike cylinder sets (a.k.a. trees), which are a special case. Ref. [46] showed that energy-norm-based sparse grids and Kolmogorov approximation schemes have bounds that are independent of d.

Ref. [47] showed that efficient approximation schemes can be developed by the use of superpositions. Every continuous function of several variables can be represented by the superposition of functions with only two variables. Ref. [20] considered discontinuous functions.

As Kolmogorov might have said—there are no true multivariate problems, only superpositions of univariate affine ones!

3.3. Kernel Smoothing: Interpolation

The theory of kernel methods was developed by Fredholm in the context of integral equations [48]. The idea is to represent a function as a linear combination of basis functions, which are called kernels.

$$f\left(x\right)={\int }_a^{b}K(x,x^{\prime })d\mu \left(x^{\prime }\right)d{x}^{\prime }\mathrm{where}x=(x_1,\dots ,x_d)$$

Here, the unknown function, $f\left(x\right)$, is represented as a linear combination of kernels, $K(x,x^{\prime })$, with unknown coefficients, $\varphi \left(x^{\prime }\right)$. The kernels are known, and the coefficients are unknown. The coefficients are found by solving the integral equation. The first work in this area was done by Abel [49], who considered equations of the form above.

Nowadays, we call those equations Volterra integral equations of the first kind. Integral equations typically arise in inverse problems. Their significance extends beyond their historical origins, as kernel methods have become instrumental in addressing one of the fundamental challenges in modern mathematics, that is, the curse of dimensionality.

Bartlett [50,51] proposed the use of kernels to estimate the regression function. The idea is to estimate the regression function, $f\left(x\right)$, at point x by averaging the values of the response variable, $y_i$, at points $x_i$ that are close to x. The kernel is used to define the weights.

The regression function is estimated as follows:

$$\widehat{f}\left(x\right)=\sum _{i=1}^d{y}_{i}K(x,x_i)/\sum _{i=1}^{d}K(x,x_i),$$

where the kernel weights are normalized.

Both Nadaraya [50] and Watson [51] considered the symmetric kernel $K(x,x^{\prime })=K(\parallel x^{\prime }-x{\parallel }_2)$, where $\left|\right|·{\left|\right|}_2$ is the Euclidean norm. The most popular kernel of that sort is the Gaussian kernel:

$$K(x,x^{\prime })=exp\left(-{\displaystyle \frac{\parallel x-x^{\prime }{\parallel }_2^2}{2{\sigma }^2}}\right).$$

Alternatively, the two-norm can be replaced by the inner product: $K(x,x^{\prime })=exp\left(x^T{x}^{\prime }/2{\sigma }^2\right)$.

Later, Ref. [52] proposed using kernels to estimate the density function. The idea is to estimate the density function $f\left(x\right)$ at point x by averaging the values of the kernel $K(x,x_i)$ at points $x_i$ that are close to x. This idea was applied in many contexts by statisticians [53,54], machine learners [55], and engineers [56]. K-GAM builds upon these foundations.

Kernel methods are supported by numerous generalization bounds, which often take the form of inequalities that describe the performance limits of kernel-based estimators. A particularly important example is the Bayes risk for k-nearest neighbors (k-NN), which can be expressed in a kernel framework as follows:

$$\widehat{f}\left(x\right)=\sum _{i=1}^N{w}_i{y}_i\mathrm{where}{w}_i:=K(x_i,x)/\sum _{i=1}^{N}K(x_i,x)$$

k-NN classifiers have been proven to converge to an error rate that is bounded in relation to the Bayes error rate, with the exact relationship depending on the number of classes. For binary classification, the asymptotic error rate of k-NN is, at most, $2R^{*}(1-R^{*})$, where $R^{*}$ is the Bayes error rate. This theoretical bound suggests potential for improvement in practice. Cover and Hart proved that interpolated k-NN schemes are consistent estimators, meaning that their performance improves with the increasing sample size.

3.4. Training Rates

Consider the non-parametric condition regression, $y_i=f\left(x_i\right)+{ϵ}_i$ where $x_i=(x_{1i},\dots ,x_{di})$. We wish to estimate $f(x_1,\dots ,x_d)$ where $x=(x_1,\dots ,x_d)\in {[0,1]}^d$. From a classical risk perspective, we define the following:

$$R(f,{\widehat{f}}_N)=E_{X,Y}\left({\parallel f-{\widehat{f}}_N\parallel }^2\right)$$

where $\parallel .\parallel$ denotes the $L^2\left(P_X\right)$-norm.

Under standard assumptions, we have an optimal minimax rate ${inf}_{\widehat{f}}{sup}_{f}R(f,{\widehat{f}}_N)$ of $O_p\left(N^{-2\beta /(2\beta +d)}\right)$ for $\beta$-Hölder-smooth functions f. This rate depends on the dimension d, which can be problematic in high-dimensional settings. By restricting the class of functions, better rates can be obtained, including ones that do not depend on d. In this sense, we avoid the curse of dimensionality. Common approaches include considering the class of linear superpositions (a.k.a. ridge functions) and projection pursuit models.

Another asymptotic result comes from a posterior concentration property. Here, ${\widehat{f}}_N$ is constructed as a regularized MAP (maximum a posteriori) estimator, which solves the optimization problem:

$${\widehat{f}}_N=arg\underset{{\widehat{f}}_N}{min}{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{f}}_N\left(x_i\right))}^2+\varphi \left({\widehat{f}}_N\right)$$

where $\varphi \left(\widehat{f}\right)$ is a regularization term. Under appropriate conditions, the ensuing posterior distribution $\Pi \left(f\right|x,y)$ can be shown to concentrate around the true function at the minimax rate (up to a $logN$ factor).

A key result in the deep learning literature provides convergence rates for deep neural networks. Given a training dataset of input–output pairs ${(x_i,y_i)}_{i=1}^N$ from the model $y=f\left(x\right)+ϵ$, where f is a deep learner (i.e., superposition of functions)

$$f=g_L\circ \dots g_1\circ g_0$$

where each $g_i$ is a ${\beta }_i$-smooth Hölder function with $d_i$ variables, we have the following condition:

$$|g_i\left(x\right)-g_i\left(y\right)|<{|x-y|}^{\beta }.$$

Then, the estimator has an optimal rate:

$$O\left(\underset{1\le i\le L}{max}{N}^{-2{\beta }^{*}/(2{\beta }^{*}+d_i)}\right)\mathrm{where}{\beta }_i^{*}={\beta }_i\prod _{l=i+1}^{L}min({\beta }_l,1)$$

This result can be applied to various function classes, including generalized additive models of the following form:

$$f_0\left(x\right)=h\left(\sum _{p=1}^d{f}_{0,p}\left(x_p\right)\right)$$

where $g_0\left(z\right)=h\left(z\right)$, $g_1(x_1,\dots ,x_d)=(f_{01}\left(x_1\right),\dots ,f_{0d}\left(x_d\right))$ and $g_2(y_1,\dots ,y_d)={\sum }_{i=1}^d{y}_i$. In this case, $d_1=d_2=1$, and assuming h is Lipschitz, we obtain an optimal rate of $O\left(N^{-1/3}\right)$, which is independent of d.

Ref. [57] shows that deep ReLU networks also have an optimal rate of $O\left(N^{-1/3}\right)$ for certain function classes. For 3-times differentiable (e.g., cubic B-splines ), Ref. [58] found a rate of $O\left(N^{-3/7}\right)=O\left(N^{-3/(2\times 3+1)}\right)$. Ref. [8] found a rate $O\left(N^{-1}\right)$ for Kolmogorov spline networks.

Finally, it is worth noting the relationship between expected risk and empirical risk. The expected risk, R, is typically bounded by the empirical risk plus a term of order $1/\sqrt{N}$:

$$R(y,f^{★})\le {\displaystyle \frac{1}{N}}\sum _{i=1}^{N}R(y_i,f^{★}\left(x_i\right))+O\left({\displaystyle \frac{\parallel f\parallel }{\sqrt{N}}}\right)$$

where $f^{★}$ is the minimizer of the expected risk. However, in the case of interpolation, where the model perfectly fits the training data, the empirical risk term becomes zero, leaving only the $O(1/\sqrt{N})$ term.

4. Transformers as Kernel Smoothing

The fundamental connection between K-GAMs and Transformers becomes clear when we decompose Transformer architectures into their core operations, that is, embedding and kernel smoothing. This decomposition reveals why K-GAMs serve as natural competitors to Transformers, as both architectures leverage these same fundamental principles, albeit through different mechanisms. In both architectures, the embedding phase transforms raw inputs into a higher-dimensional space, where relationships can be more easily captured, while the smoothing phase aggregates information across this embedded space to produce outputs.

Transformers have become a main building block for various natural language processing (NLP) tasks and have been extended to other domains as well due to their effectiveness. The Transformer architecture is primarily designed to handle sequential data, making it well-suited for tasks such as machine translation, language modeling, text generation, and more.

The Transformer approach to embedding begins by converting input tokens into vectors in a high-dimensional space, augmented with positional information to preserve sequential structure. K-GAMs, in contrast, employ the Köppen function to transform inputs into a universal embedding space. This difference in embedding strategy reflects a fundamental trade-off, that is, Transformers learn task-specific embeddings that can adapt to particular domains, while K-GAMs leverage a universal embedding that can theoretically capture any continuous function.

Next, we will discuss the role of kernel smoothing in Transformers. Ref. [59] first introduced kernel smoothing for sequence-to-sequence learning through what became known as the attention mechanism. This approach estimates the probability of the next word in the sequence using a so-called context vector $c_i$, which is a weighted average of the vectors from the input sequence $h_j$:

$$c_i=\sum _{j=1}^n{\alpha }_{ij}{h}_j,$$

where ${\alpha }_{ij}$ denotes the weights. The weights are defined by a kernel function, specifically a normalized exponential (softmax):

$${\alpha }_{ij}={\displaystyle \frac{exp\left(e_{ij}\right)}{{\sum }_{k=1}^{n}exp\left(e_{ik}\right)}}.$$

When used for self-supervised learning, this attention mechanism is called self-attention. When a sequence is mapped to a matrix M, it is called multi-head attention.

This formulation mirrors the structure of traditional kernel smoothing methods, with the key innovation being that, rather than using a fixed similarity measure like Euclidean distance, the energy function $e_{ij}=a(s_{i-1},h_j)$ is learned through a neural network. This neural network adaptively measures the similarity between the last generated element of the output sequence $s_{i-1}$ and the j-th element of the input sequence $h_j$, allowing the model to learn context-dependent attention patterns.

Both architectures use these phases to reduce the dimensionality of the learning problem while preserving the ability to capture complex relationships. The key difference lies in how they implement these phases; Transformers learn both the embedding and smoothing operations, while K-GAMs use a fixed, universal embedding (Köppen function) followed by learned smoothing.

The main advantage of using smoothing techniques is that they are parallelizable. This is the key draw of Transformers. Current language models such as BERT, GPT, and T5 rely on the Transformer approach. Further, Transformers have also been applied to computer vision and other domains. Their ability to capture long-range dependencies and their scalability have made them powerful tools for a wide range of applications. See Ref. [60] for further details.

5. Application

5.1. Simulated Data

We also apply the K-GAM architecture to a simulated dataset to evaluate its performance on data with known structures and relationships. The dataset contains 100 observations generated from the following function:

$$\begin{array}{cc}\hfill & y=\mu \left(x\right)+ϵ,ϵ\sim \mathcal{N}(0,1)\hfill \\ \hfill & \mu \left(x\right)=10sin\left(\pi x_1{x}_2\right)+20{(x_3-0.5)}^2+10x_4+5x_5.\hfill \end{array}$$

The goal is to predict the function $y\left(x\right)$ based on the input x. The dataset is often used as a benchmark dataset for regression algorithms due to its diverse mix of relationships (linear, quadratic, nonlinear, Gaussian random noise) between the input features and the target function. The plot of $\mu$ (no noise) vs. y (noise) is shown in Figure 2.

We use the Köppen function to transform the five-dimensional input into a set of 11 features ($2d+1$). We then learn the outer function g using a ReLU network. To thoroughly investigate the model’s capabilities, we implement two distinct approaches to learning the outer function. The first approach uses different g functions for each feature, following the original KST formulation. This allows each function to specialize in capturing specific patterns, but it might be more difficult to train and has more parameters. The second approach uses a single g function for all features, as proposed by [61], providing a more unified and parameter-efficient representation.

Figure 3 illustrates these architectural choices in detail, showing how the information flows through each version of the model. For the first model with multiple $g_i$ functions, the dimensions of each $g_i$ are as follows: $W_i^0\in {\mathbb{R}}^{16\times 1}$ and for $j=1,\dots ,18$, $W_i^j\in {\mathbb{R}}^{16\times 16}$. Exemplary outer functions are shown in Figure 4.

The next architecture, which uses only one function, g, for all features, maintains a similar structure to the multiple-g function approach. The only difference is in the dimensionality of the inner layers—we increase the width from 16 to 200. This increased capacity allows the single function to learn more complex patterns and compensate for the constraint of using just one function instead of multiple specialized ones. The behavior of this unified outer function is shown in Figure 5, where we can observe how it adapts to handle multiple feature transformations simultaneously.

5.2. Iris Data

We apply the KST architecture to an iris dataset. The iris dataset is a classic dataset in machine learning and statistics. It contains 150 observations of iris flowers. Each observation contains four features, namely, sepal length, sepal width, petal length, and petal width. The goal is to predict the species of the iris flower based on these features. The dataset contains three classes of iris flowers, namely, setosa, versicolor, and virginica. The dataset is often used as a benchmark dataset for classification algorithms. The dataset has five variables, which include four characteristics of the iris flower and the species of the flower. Figure 6 shows the scatter plots of the iris dataset.

We calculate the mean ${\mu }_{\mathrm{SL}}$ of the sepal length column and use a binary variable $y_i=\mathrm{Sepal}{\mathrm{Length}}_i>{\mu }_{\mathrm{SL}}$ as the output. We use the other three flower characteristics, $x_1,x_2,x_3$, as input variables. We use a classical GAM model to fit the data and compare it to KST-GAN. The classical generalized additive model is given by the following:

$$\mathrm{logit}\left(P(y=1\mid x)\right)={\beta }_0+f_1\left(x_2\right)+f_2\left(x_3\right)+f_3\left(x_4\right),$$

where $f_1,f_2,f_3$ are smooth functions of the input features. The KST-GAN model is given by the following:

$$\mathrm{logit}\left(P(y=1\mid \psi )\right)={\beta }_0+f_1\left({\psi }_1\right)+\dots +f_7\left({\psi }_7\right),$$

where ${\psi }_q$ is the Köppen transformation function (in Appendix A) of the input features

$${\varphi }_q(x_1,x_2,x_3)=\sum _{i=1}^3{\alpha }_i\psi (x_i+aq),q=1,\dots ,7.$$

We use $m=2d+1=7$ and $k=6$ to transform the input features into a set of seven features. We then learn the outer function g using a classical GAN approach. We use the mgcv package in R to fit the GAM model. This package uses a penalized likelihood approach to fit the model [62].

Table 1. Summary of the classical GAM, KST GAM, and GLM models applied to the iris dataset.

	Classical GAM	KST GAM	GLM
(Intercept)	−1.758	18.064	−28.339
	(1.435)	(54.091)	(8.371)
Sepal.Width			2.669
			(1.499)
Petal.Length			6.377
			(2.090)
Petal.Width			−5.105
			(2.292)
Num.Obs.	105	105	105
R2	0.790	0.591
AIC	43.0	207.3	50.6
BIC	59.3	258.1	61.2
Log.Lik.			−21.281
RMSE	0.22	0.29	0.26

compares the performances of the GAM and KST-GAM models. We also include a classical logistic regression model for comparison. The KST-GAM model has a higher AIC and BIC compared to the GAM model. The KST-GAM model has a comparable RMSE.

Table 2. Confusion out-of-sample matrix for the GAM model applied to the iris dataset.

	Predicted 0	Predicted 1
Actual 0	21	1
Actual 1	2	21

and

Table 3. Confusion out-of-sample matrix for the KST GAM model applied to the iris dataset.

	Predicted 0	Predicted 1
Actual 0	19	3
Actual 1	8	15

show the confusion matrices for the GAM and KST GAM models, respectively. The KST GAM model has a lower accuracy compared to the GAM model.

Figure 7 shows the original features plotted against those fitted by the GAM function for both original inputs and the KST-transformed inputs.

Overall, the iris example demonstrates the inability of the GAM model to capture the complex relationships between the transformed features and the target variable.

The comparison between traditional GAM and K-GAM approaches in Figure 7 shows some notable differences. The GAM plots show slightly smoother relationships between individual features and the output. For instance, petal width and petal length show particularly strong linear relationships with the target variable. In comparison, the K-GAM architecture seems to capture different aspects of the feature relationships. Notably, while one of the features retains a largely linear relationship, the other two features encode more complicated relationships. This difference demonstrates the effect of the Köppen function’s mapping.

Next, the simulated data study explores how K-GAM handles data with known structures, as well as the differences between using multiple outer functions and a single shared outer function within the K-GAM framework. The comparison between multiple $g_i$ functions (Figure 4) and a single g function (Figure 5) demonstrates the architectural flexibility of the approach. The single g function variant shows a highly variable pattern across its input range, showing that the increased dimension compensates for the reduced flexibility of having a single function by developing more complex internal representations. The ability to choose between multiple specialized functions or a single shared function provides a useful degree of freedom in the model’s design.

5.3. Image Processing

A number of authors have proposed the use of Kolmogorov networks for image processing tasks (for example, see Refs. [1,2]). More recently, functional KANs were proposed in [12]. Figure 8 shows the error outputs for the MNIST dataset. Equivalent rates for the MNIST dataset can be found in [63].

Functional KAN methods have been shown to be effective for image processing tasks and have been shown to be promising in other tasks. Ref. [8] provided theoretical results for Kolmogorov networks applied to image segmentation tasks. Another alternative is deep partial least squares (DPLS) networks—a type of Kolmogorov network that uses partial least squares regression to learn the network weights [64].

6. Discussion

At its core, our work reinforces Kolmogorov’s profound insight, that is, there are no true multivariate problems, only superpositions of univariate affine ones. This principle guides our approach to efficiently decomposing complex multivariate functions into simpler univariate components through the Köppen transformation. The results demonstrate that while K-GAM can effectively model both real (iris) and simulated data, the internal representations it learns may be more complex than traditional GAM approaches. The approach appears particularly effective at capturing nonlinear patterns, although the interpretability of individual feature effects becomes more challenging due to the Köppen function transformation. Importantly, the K-GAM approach requires significantly fewer parameters compared to standard GAMs, as it leverages a shared embedding space through the Köppen function and can capture nonlinear relationships without requiring explicit interaction terms. However, most of the research since 2010 has been about increasing the depth of the networks and not about the width of the networks. The K-GAM approach is a step forward in the direction of increasing the width of the networks. The difference is that the Kolmogorov architecture is based on non-smooth functions, while the deep learning architecture is based on the composition of smooth functions. As shown by [65], it is impossible to obtain a universal approximation using a two-layer architecture with smooth functions.

Our findings suggest several promising directions for future research across both theoretical and practical domains. One priority involves the enhancement of the scalability of KST-based approaches, along with the characterization of the function classes for which K-GAM performs optimally over existing alternatives. Given K-GAM’s parameter efficiency, exploring Bayesian learning methods could provide principled approaches to uncertainty quantification and model regularization. An important optimization for K-GAM would be a specialized optimization algorithm capable of handling the discontinuities inherent in the Köppen function and the development of an efficient training algorithm specifically designed for high-dimensional problems.

These findings suggest that K-GAM networks represent a promising direction for efficient function approximation, particularly in scenarios where traditional deep learning approaches may be computationally intractable or parameter-inefficient. The combination of theoretical guarantees from Kolmogorov’s superposition theorem with modern machine learning techniques opens new avenues for both theoretical research and practical applications.