Universal Approximation of Operators with Transformers and Neural Integral Operators

Abstract

We study the universal approximation properties of transformers and neural integral operators for operators in Banach spaces. In particular, we show that the transformer architecture is a universal approximator of integral operators between Hölder spaces. Moreover, we show that a generalized version of neural integral operators, based on the Gavurin integral, treats them as universal approximators of arbitrary operators between Banach spaces. Lastly, we show that a modified version of the transformer, which uses Leray–Schauder mappings, is a universal approximator of operators between arbitrary Banach spaces.

1. Introduction

Operator learning constitutes a branch of deep learning dedicated to approximating nonlinear operators defined between Banach spaces. The appeal of this field stems from its capacity to model complex phenomena, such as dynamical systems, for which underlying governing equations remain elusive. Operator learning is thought to have originated with the theoretical groundwork established in [1] and subsequently implemented in [2].

Operator learning has been successfully applied in several fields of science, including learning neural integro-differential equations [3], integral operators [4,5], quantum state tomography [6], quantum circuit learning [7], control theory [8], partial differential equations [9], and holographic quantum chromodynamics [10], to mention but a few.

Transformers [11] have emerged as a powerful tool in natural language processing, and they have more recently been employed for addressing problems in operator learning, including dynamical systems. Previous research has explored their use in integral equations [4,5], inverse problems [12], fluid flows [13], and PDEs [14], among others.

Recent theoretical work highlights the strong approximation capabilities of transformer architectures across a range of settings. In [15], the authors showed that a variation in transformers’ attention mechanism, which includes free parameters, allows the recovery of polynomials with zero error over a compact subset of a Euclidean space. This procedure has the drawback of potential exponential scaling of free parameters. It was shown in [16] that transformers can approximate sequence-to-sequence functions with respect to a suitable “average” error. The case of approximating sequence-to-sequence functions with some smoothness and with infinite dimensional inputs was studied in [17], where learnable positional encoding was employed. More recently, in [18], these works were further extended to show that sequence-to-sequence functions with Hölder and Sobolev regularity can be approximated in the $L^p$ norm.

Recent studies have explored the application of transformers to operator modeling. For instance, [19] demonstrated the efficacy of attention mechanisms in learning operators with low regularity, specializing in Izhikevich and tempered fractional LIF neuron models. This work builds upon previous research [20,21] that established near-optimal error bounds for learning nonlinear operators between infinite-dimensional Banach spaces, notably DeepONets [22]. In addition, the work of [19] shows that on Banach spaces with the approximation property, transformers with positional encoding are universal approximators, using the addition of an encoder and a decoder and applying the results of [23] to uniformly approximate continuous functions via CNNs.

Transformers without positional encoders and without additional encoders and decoders have not yet been extensively explored in the case of operator learning problems. In fact, the results of [16,17,18], when applied directly to the proof of [19] in this setting, give error bounds that are either not uniform or are restricted to some classes of functions that are not directly related to the construction in [19].

The main results of the present paper are the following. We establish a universal approximation framework for Urysohn integral operators between Hölder spaces using transformer architectures without positional encoding, relying instead on their permutation-equivariant approximation properties. Our methods can also be applied for any equivariant universal approximator of sequence-to-sequence mappings, of which the transformer without positional encoding is an example. Moreover, incorporating Leray–Schauder mappings as encoders, we obtain a universal approximation result for (nonlinear) operators between arbitrary Banach spaces, including those without the approximation property. Lastly, we introduce a generalized neural integral operator based on the Gavurin integral and prove its universal approximation property for operators between Banach spaces under some smoothness assumption.

This paper is organized as follows. Section 2 outlines the necessary preliminaries. In Section 3, we prove that transformers are universal approximators for Hölder spaces. Section 4 investigates the universal approximation capabilities of neural integral operators for operators in Banach spaces. We conclude with Section 5, exploring further perspectives and open questions.

2. Preliminaries

2.1. Notation

Throughout this article, we assume a discretization of the interval $[a,b]$ as $a=t_0<t_1<\cdots <t_{n-1}<t_n=b$. Unless otherwise explicitly stated, over any subinterval, i.e., $[a,t_1],\cdots ,[t_{n-1},b]$, the point $x_i\in [t_i,t_{i+1}]$.

Let $X=C^{k,\alpha }\left([a,b]\right)$ denote the Hölder space of k-differentiable functions with the $k^{\mathrm{th}}$ derivative that are Hölder continuous. We let $T:X⟶X$ be an integral operator. Consider the values $x_i$ and $z_k=y\left(t_k\right)$ for $i,k=1,\dots ,n$, where $x_i$ and $t_k$ vary among the points of the discretization determined by the integration approximation.

Let the kernel of T be the function $G:\mathbb{R}\times [a,b]\times [a,b]⟶\mathbb{R}$, which is continuous with respect to all variables, and is continuously differentiable with respect to the first and third variables. Here, G takes as input a function u and two “time” variables from the interval $[a,b]$. Then, T is defined according to the assignment

$$\begin{array}{c}\hfill T\left(u\right)\left(x\right)={\int }_a^{b}G(u\left(t\right),x,t)dt.\end{array}$$

In other words, T is an Urysohn integral operator.

For a function $f:{\mathbb{R}}^n⟶{\mathbb{R}}^m$, we indicate by $f^i(x_1,\dots ,x_n)$ the $i^{\mathrm{th}}$-entry of the vector $f(x_1,\dots ,x_n)$.

Following the previous conventions, for each $i=1,\dots ,n$ in the rest of the paper, we indicate

$$\begin{array}{c}\hfill T^i\left(u\right)(x_1,\dots ,x_n):=\int G(u\left(t\right),x_i,t)dt,\end{array}$$

where, as mentioned above, $x_i\in [t_i,t_{i+1}]$ for each i. So, $T\left(u\right)$ is the vector function whose $i^{\mathrm{th}}$-entry function is $T^i\left(u\right)$. Given a function $F:{\mathbb{R}}^n⟶{\mathbb{R}}^n$, we introduce the following norm

$$\begin{array}{c}\hfill {\parallel F\parallel }_p={\left({\displaystyle \sum _{i=1}^n}{\int }_{[a,t_1]\times \cdots \times [t_{n-1},b]}{\left|F^i(x_1,\dots ,x_n)\right|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}},\end{array}$$

(1)

which corresponds to the Euclidean norm of the component-wise p-norm, and can be applied to $T^i\left(u\right)$ as a particular case. This p-norm will be the objective of the approximation results over Hölder spaces for transformers.

2.2. Leray–Schauder Mappings

Leray–Schauder mappings [24,25] are a fundamental tool in nonlinear functional analysis. They are used to establish the existence of solutions to nonlinear problems by extending the concept of degrees from finite-dimensional spaces to infinite-dimensional Banach spaces. Roughly speaking, given a compact set K and a finite dimensional subspace of a Banach space, we want to find a continuous mapping onto the finite subspace satisfying the property that the norm of elements in the compact set and the image of the mapping is smaller than a fixed $ϵ>0$. Following [25], we define Leray–Schauder mappings on a Banach space X as follows. Let K be a compact set, and $E_n$ an n-dimensional space spanned by the $ϵ$-net $x_1,\dots ,x_n$ of K. We define $P_n:X⟶E_n$ by the assignment

$$\begin{array}{c}\hfill P_{n}x={\displaystyle \frac{{\sum }_{i=1}^n{\mu }_i\left(x\right)x_i}{{\sum }_{i=1}^n{\mu }_i\left(x\right)}},\end{array}$$

where

$$\begin{array}{c}\hfill {\mu }_i\left(x\right)=\left\{\begin{array}{c}ϵ-\parallel x-x_i\parallel ,\parallel x-x_i\parallel \le ϵ\hfill \\ 0,\parallel x-x_i\parallel >ϵ\hfill \end{array}\right.\end{array}$$

(2)

for all $i=1,\dots ,n$. One sees directly that this mapping satisfies the property that

$$\begin{array}{c}\hfill \parallel x-P\left(x\right)\parallel <ϵ,\end{array}$$

for all $x\in K$.

2.3. Gavurin Neural Integral Operator $\mathfrak{T}\left(x\right)$

We hereby recall the definition of the Gavurin integral [26], and use this to define a class of neural integral operators of relevance for the rest of this paper.

We follow [27], Chapter 17, and consider a mapping $R:[x_0,\overline{x}]\subset X\to B(X,Y)$, where $B(X,Y)$ denotes the space of bounded linear operators from X to Y, two Banach spaces. Let $0=t_0<t_1<\cdots <t_n=1$ be a partition of $[0,1]$, and let ${\tau }_k\in [t_k,t_{k+1}]$. Then, the integral of R is defined in a procedure similar to the classical Riemann integral as

$$\begin{array}{c}\hfill {\int }_{x_0}^{x_0+\Delta x}R\left(x\right)dx={\int }_0^{1}R(x_0+t\Delta x)\Delta xdt=\underset{n\to \infty }{lim}{\displaystyle \sum _{k=0}^{n-1}}R(x_0+{\tau }_k\Delta x)\Delta x(t_{k+1}-t_k),\end{array}$$

(3)

when the limit exists and is independent of the choices of ${\tau }_k$. We indicate the action of the integral operator on an element z of X as ${\int }_{x_0}^{x_0+\Delta x}R\left(x\right)\left(z\right)dx$.

We define a Gavurin integral operator to be an integral operator $T:X⟶Y$, which is the point-wise convex sum of the integral operators, as in Equation (3), as follows:

$$\begin{array}{ccc}\hfill T\left(z\right)& =& \sum _i{\eta }_i\left(z\right)[F_i\left(z\right)+{\int }_{x_0}^{x_0+\Delta x}{R}_i\left(x\right)\left(z\right)dx],\hfill \end{array}$$

(4)

where for every z, ${\sum }_i{\eta }_i\left(z\right)=1$, i.e., the coefficients are a partition of unity, and $F_i$ values are arbitrary linear operators.

We define a Gavurin neural integral operator to be an operator approximating the form in (4) with a Riemann sum consisting of Leray–Schauder maps and neural networks in the following sense. Let P and $P^{\prime }$ denote two Leray–Schauder mappings, $P:U⟶X$ and $P^{\prime }:U^{\prime }⟶Y$, where U and $U^{\prime }$ are open sets containing two given compacts, $\mathbb{K}$ and ${\mathbb{K}}^{\prime }$. Then, $P\left(\mathbb{K}\right)$ and $P^{\prime }\left({\mathbb{K}}^{\prime }\right)$ both span the finite dimensional subspaces E and $E^{\prime }$ of X and Y, respectively, by construction. We let $\phi$ and ${\phi }^{\prime }$ denote two linear homeomorphisms $\phi :E⟶{\mathbb{R}}^n$ and ${\phi }^{\prime }:E^{\prime }⟶{\mathbb{R}}^m$, where we have set $dimE=n$ and $dim{E}^{\prime }=m$. Let $f:{\mathbb{R}}^n⟶{\mathbb{R}}^m$ be a neural network, and consider the composition $F_{P,f}:={\phi }^{\prime -1}f\phi P$. Then, a Gavurin neural integral operator is a neural network defined as

$$\begin{array}{ccc}\hfill F_{P,f_1,\dots ,f_q,g_1,\dots ,g_q}\left(z\right)& :=& \sum _i{\eta }_i\left(z\right)F_{P,f_i}\left(z\right)+\sum _r\sum _i\sum _k{\eta }_i\left(z\right){\alpha }_k\left(t_r\right)F_{P,g_i}\left(z\right),\hfill \end{array}$$

where ${\left\{\eta \right\}}_i$ is a partition of unity subordinate to an open cover of $\mathbb{K}$, ${\left\{{\alpha }_k\right\}}_k$ is a partition of unity of $[0,1]$, and $0=t_0\le \dots \le t_q=1$ is a discretization of the interval $[0,1]$. All sums in the definition are finite, and $F_{P,f_1,\dots ,f_q,g_1,\dots ,g_q}$ implicitly depends on the chosen discretization of $[0,1]$, as well as the partitions of unity ${\left\{{\eta }_i\right\}}_i$ and ${\left\{{\alpha }_k\right\}}_k$. We notice that the sum over r is substantially mimicking a Riemann sum, therefore approximating an integral.

2.4. Models with Permutation-Equivariance

Equivariance can be formalized using the concept of a G-set for a group G consisting of a set S and where G acts on S from the right. Given two G-sets, X and Y, for the same group G, a function $f:X\to Y$ is said to be G-equivariant if $f(x·g)=f\left(x\right)·g$ for all $g\in G$ and all $x\in X$.

A transformer is a sequence-to-sequence model composed of an encoder and a decoder, both built from stacks of self-attention and feedforward layers. A transformer defines a function $\mathrm{T}:{\mathcal{X}}^n\to {\mathcal{Y}}^m,$ where $\mathcal{X}$ and $\mathcal{Y}$ are the input and output token spaces, respectively, and n, m are the input and output sequence lengths. Given an input sequence of tokens $x=(x_1,\dots ,x_n)$, each token is mapped to a vector via an embedding function ${\mathbf{e}}_i=\mathrm{Embed}\left(x_i\right)\in {\mathbb{R}}^d.$ A positional encoding ${\mathbf{p}}_i\in {\mathbb{R}}^d$ is added to retain order information ${\mathbf{z}}_i^{\left(0\right)}={\mathbf{e}}_i+{\mathbf{p}}_i.$ At each layer ℓ, the input ${\mathbf{Z}}^{(\mathsf{\ell }-1)}=({\mathbf{z}}_1^{(\mathsf{\ell }-1)},\dots ,{\mathbf{z}}_n^{(\mathsf{\ell }-1)})$ is transformed via multi-head self-attention, as follows:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)V,$$

where $d_k$ is the length of the key vectors, used as a scaling factor to keep numbers stable,

$${\mathbf{z}}_i^{(\mathsf{\ell },\mathrm{attn})}=\mathrm{MultiHead}\left({\mathbf{z}}_i^{(\mathsf{\ell }-1)}\right)=\mathrm{Concat}(h_1,\dots ,h_H)W^O,$$

where each head $h_j$ is

$$h_j=\mathrm{Attention}(Q_j,K_j,V_j),Q_j={\mathbf{Z}}^{(\mathsf{\ell }-1)}{W}_j^Q,K_j={\mathbf{Z}}^{(\mathsf{\ell }-1)}{W}_j^K,V_j={\mathbf{Z}}^{(\mathsf{\ell }-1)}{W}_j^V.$$

Each position i is updated via a position-wise feedforward network:

$${\mathbf{z}}_i^{(\mathsf{\ell },\mathrm{ff})}=\mathrm{FFN}\left({\mathbf{z}}_i^{(\mathsf{\ell },\mathrm{attn})}\right)=\sigma ({\mathbf{z}}_i{W}_1+b_1)W_2+b_2.$$

The full update rule at each layer ℓ is

$$\begin{array}{cc}\hfill {\mathbf{z}}_i^{(\mathsf{\ell },\mathrm{norm}1)}& =\mathrm{LayerNorm}\left({\mathbf{z}}_i^{(\mathsf{\ell }-1)}+{\mathbf{z}}_i^{(\mathsf{\ell },\mathrm{attn})}\right),\hfill \\ \hfill {\mathbf{z}}_i^{(\mathsf{\ell })}& =\mathrm{LayerNorm}\left({\mathbf{z}}_i^{(\mathsf{\ell },\mathrm{norm}1)}+{\mathbf{z}}_i^{(\mathsf{\ell },\mathrm{ff})}\right).\hfill \end{array}$$

In the context of transformers, particularly in machine learning models, equivariance refers to equivariance with respect to the natural symmetric group action on sequences of inputs–outputs. More specifically, for the function $f:X\to Y$ (e.g., a neural network layer), and the group G acting on both X and Y, the function f is said to be equivariant with respect to G if

$$f(x·g)=f\left(x\right)·g,\forall g\in G,x\in X,$$

where $x·g$ and $f\left(x\right)·g$ denote the actions of g on X and Y, respectively.

Deep Sets [28] are explicitly designed to be permutation-invariant or -equivariant by applying a shared function to each element, followed by a symmetric aggregation operation (e.g., summation). Neural Functional models [29] generalize this idea by treating input data as functions over domains and ensure equivariance under transformations such as permutations by construction. Transformers [11], before the addition of positional encodings, are naturally permutation-equivariant due to the symmetric structure of the self-attention mechanism, i.e., each token tends to all others through shared attention weights, making the output invariant to the order of inputs under shared weights. However, once positional information is introduced, this permutation symmetry is typically broken.

The approach pursued in this article can be applied to any architecture that is equivariant and a universal approximator of sequence-to-sequence mappings, in the norm defined in (1). In other words, given an approximator of sequence-to-sequence mappings, we can show that there is a corresponding construction that is a universal approximator of nonlinear operators between the Hölder spaces. We pose the following useful definition.

In this article, we will use the term generalized transformer to denote a mapping $\mathcal{W}$ defined on a space of functions as follows. We have a locally finite partition of unity $\left\{{\eta }_i\right\}$ defined on the space subordinate to some open covering $\mathcal{U}$, and we let $W_U$ denote a standard transformer for each open U in the covering $\mathcal{U}$, whether it has positional encoding or not. For a function $f\in U$, $W_U\left(f\right)$ is defined by applying f to some pre-defined sampling scheme of the domain of the space of functions, and then applying $W_U$ on the corresponding outputs of f. We then set

$$\begin{array}{ccc}\hfill \mathcal{W}\left(f\right)& =& \sum _i{\eta }_i\left(f\right)W_U\left(f\right).\hfill \end{array}$$

Each sum is finite since the partition of unity is locally finite. Details of this procedure are provided below in Section 3.

2.5. Notation and Generalities

The notations used in the following sections are summarized in Table 1. The precise definitions of each notation are given in the main text and are used throughout this paper.

Table 1. Summary of notations used in the rest of the manuscript.

Notation	Description
T	The mapping between Banach spaces to be approximated
$\mathcal{T}$	The integration scheme approximating an integral operator T
$T^i$(u)	$i^{\mathrm{th}}$-component of the function induced by evaluating the operator T at u
$\mathfrak{T}$	Leray–Schauder transformer or Gavurin neural integral operator
$W_y$	Transformer architecture associated to a function y
$\mathcal{W}$	Transformer architecture obtained via partition of unity
F	A real vector function $F:{\mathbb{R}}^n\times {[a,b]}^n\to {\mathbb{R}}^n$
$F_y$	Function ${[a,b]}^n\to {\mathbb{R}}^n$ induced by F and y
$F_y^i$	$i^{\mathrm{th}}$-component of $F_y$

In this work, the term operator will refer to any continuous mapping, without restriction of linearity. This follows the common notations in operator learning, where nonlinear maps between function spaces are also called operators.

3. Universal Approximation for Transformers

We recall the following definitions. Let $f:[a,b]⟶\mathbb{R}$ be a function. Then, we say that f is Hölder continuous if there exists a constant c such that

$$\begin{array}{ccc}\hfill \left|f\right(x)-f(y\left)\right|& \le & {c|x-y|}^{\alpha },\hfill \end{array}$$

for all $x,y\in [a,b]$, where $\alpha \in (0,1]$. We define the set $C^{k,\alpha }\left([a,b]\right)$ of all Hölder functions whose kth derivative is Hölder continuous, and define the norm

$$\begin{array}{ccc}\hfill {\parallel f\parallel }_{k,\alpha }& =& {\parallel f\parallel }_k+sup\{{\displaystyle \frac{|f^k\left(x\right)-f^k\left(y\right)|}{{|x-y|}^{\alpha }}}|x,y\in [a,b],x\ne y\},\hfill \end{array}$$

where ${\parallel f\parallel }_k:={max}_{r\le k}{\parallel f^r\parallel }_{\infty }$ is the maximum among the uniform norms of the derivatives of f. The space $C^{k,\alpha }\left([a,b]\right)$ is a Banach space. In this section, we consider $C^{k,\alpha }\left([a,b]\right)$, where $1\le k\in \mathbb{N}$ and $\alpha \in (0,1]$, and let $T:C^{k,\alpha }\left([a,b]\right)⟶C^{k,\alpha }\left([a,b]\right)$ be a given integral operator. We want to show that there is a transformer architecture $\mathcal{W}$ that approximates T with arbitrarily high precision in the following sense. For an arbitrary choice of $ϵ>0$, there exist a discretization $a=t_0<t_1<\cdots <t_{n-1}<t_n=b$ of $[a,b]$ and a transformer architecture $\mathcal{W}$ such that for each $u\in C^{k,\alpha }\left([a,b]\right)$, u in some fixed set, we have that

$$\begin{array}{c}\hfill {\left({\displaystyle \sum _{i=1}^n}{\int }_{[a,t_1]\times \cdots \times [t_{n-1},b]}{|T_u^i(x_1,\dots ,x_n)-{\mathcal{W}}_u^i(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}<ϵ,\end{array}$$

(5)

where $d\mu$ is a shorthand notation for $d{x}_1\cdots d{x}_n$.

We will need several lemmas before obtaining the main result.

We set X to be a Hölder space $C^{k,\alpha }\left([a,b]\right)$ (with $k\ge 1$) of functions defined on some closed interval $[a,b]$. Let $T:X⟶X$ denote an integral operator, where we assume that T is an Urysohn integral operator and that the kernel of T is a function $G:\mathbb{R}\times [a,b]\times [a,b]⟶\mathbb{R}$ that is continuous with respect to all variables and is continuously differentiable with respect to the first and third variables. We let ${\mathbb{B}}_{\rho }$ be a ball centered at the origin, with radius $\rho$ fixed. In this article, we will consider only numerical integration schemes that are non-adaptive and that are given by continuous formulas with respect to the sampled points.

Lemma 1.

Let T and X be as above. Then, we can find an integration scheme, $\mathcal{T}$, such that

$$\begin{array}{c}\hfill {\parallel T\left(y\right)-\mathcal{T}\left(y\right)\parallel }_{\infty }<ϵ,\end{array}$$

(6)

for all $y\in {\overline{\mathbb{B}}}_{\rho }$, where ${\parallel ·\parallel }_{\infty }$ denotes the uniform norm.

Proof. 

We denote by $G:\mathbb{R}\times [a,b]\times [a,b]⟶\mathbb{R}$ the kernel of the integral operator T. Therefore,

$$T\left(y\right)\left(x\right)={\int }_a^{b}G(y\left(t\right),x,t)dt,$$

for all $x\in [a,b]$. For $y\in {\overline{\mathbb{B}}}_{\rho }$, from $\parallel y\parallel \le \rho$, it follows that ${sup}_{[a,b]}\left|y\left(x\right)\right|\le \rho$. Therefore, $G\left(y\right(t),x,t)$ takes values $y\left(t\right)$ in $[-\rho ,\rho ]$. Let $M_1$ denote the value

$$\begin{array}{c}\hfill M_1:=\underset{[-\rho ,\rho ]\times [a,b]\times [a,b]}{max}\left|{\partial }_{y}G(y\left(t\right),x,t)\right|,\end{array}$$

and let

$$\begin{array}{c}\hfill M_2:=\underset{[-\rho ,\rho ]\times [a,b]\times [a,b]}{max}\left|{\partial }_{t}G(y\left(t\right),x,t)\right|.\end{array}$$

Let us choose a quadrature rule with n points ${\left\{q_i\right\}}_{i=1}^n$ selected from $[a,b]$ to approximate the integral operator T. We indicate the quadrature by $\mathcal{T}$: $T\left(y\right)\left(x\right)={\int }_a^{b}G(y\left(t\right),x,t)dt\approx {\sum }_{i=1}^{n}G(y\left(q_i\right),x,q_i)w_i=:\mathcal{T}\left(y\right)\left(x\right)$, where the coefficients $w_i$ are the weights of the quadrature, and $G(y\left(q_i\right),x,q_i)$ is continuous with respect to x. Also, $G\left(y\right(t),x,t)$ is differentiable with respect to the variable t of an integration with a derivative given by

$$\begin{array}{c}\hfill {\displaystyle \frac{dG\left(y\right(t),x,t)}{dt}}={\partial }_{t}G(y\left(t\right),x,t)+{\partial }_{y}G(y\left(t\right),x,t)·y^{\prime }\left(t\right),\end{array}$$

for any chosen $y\in {\overline{\mathbb{B}}}_{\rho }$. For the sake of simplicity, we use the forward rectangle method in what follows, but a similar procedure (upon suitably changing the error bounds and assuming that the index k of the Hölder space is large enough) can be performed for different integration rules, such as the Midpoint, Trapezoidal, corrected Trapezoidal, Cavalieri–Simpson, Boole, etc. It is known that the error in performing the numerical integration of ${\int }_a^{b}g\left(t\right)dt$ is bounded by ${\displaystyle \frac{{(b-a)}^2}{2n}}{sup}_{[a,b]}\left|g^{\prime }\left(t\right)\right|$ [30]. Therefore, for each fixed $x_0\in [a,b]$, we have the bound

$$\begin{array}{ccc}\hfill |T\left(y\right)\left(x_0\right)-\mathcal{T}\left(y\right)\left(x_0\right)|& \le & {\displaystyle \frac{{(b-a)}^2}{2n}}\underset{[a,b]}{sup}|{\partial }_{t}G(y\left(t\right),x_0,t)+{\partial }_{y}G(y\left(t\right),x_0,t)·y^{\prime }\left(t\right)|\hfill \\ & \le & {\displaystyle \frac{{(b-a)}^2}{2n}}[\underset{[a,b]}{sup}|{\partial }_{t}G(y\left(t\right),x_0,t)|+\underset{[a,b]}{sup}|{\partial }_{y}G(y\left(t\right),x_0,t)·y^{\prime }\left(t\right)|]\hfill \\ & \le & {\displaystyle \frac{{(b-a)}^2}{2n}}(M_1\rho +M_2),\hfill \end{array}$$

from which we derive $|T\left(y\right)\left(x\right)-\mathcal{T}\left(y\right)\left(x\right)|\le {\displaystyle \frac{{(b-a)}^2}{2n}}(M_1\rho +M_2)$ for all $y\in {\overline{\mathbb{B}}}_{\rho }$ and all $x\in [a,b]$. Upon choosing $n>{(b-a)}^2{\displaystyle \frac{M_1\rho +M_2}{2ϵ}}$, we find that ${\parallel T\left(y\right)-\mathcal{T}\left(y\right)\parallel }_{\infty }<ϵ$ uniformly in ${\overline{\mathbb{B}}}_{\rho }$. □

A variation in the following argument appears also in [4,5]. We provide here a more formal approach. Before stating and proving the result, we pose the following definition.

Definition 1.

Let $\sigma$ denote a permutation of n elements, i.e., $\sigma \in {\Sigma }_n$, where ${\Sigma }_n$ is the permutation group on n elements. Then, $\sigma$ acts on the tuples of type $(z_1,\dots ,z_n,x_1,\dots ,x_n)\in {\mathbb{R}}^{2n}$ through the induced diagonal permutation representation, which just reorders the entries $z_1,\dots ,z_n$ and $x_1,\dots ,x_n$ according to the permutation $\sigma$, as follows:

$$(z_1,\dots ,z_n,x_1,\dots ,x_n){·}_1\sigma :=(z_{\sigma \left(1\right)},\dots ,z_{\sigma \left(n\right)},x_{\sigma \left(1\right)},\dots ,x_{\sigma \left(n\right)}).$$

We indicate by ${·}_2$ the permutation action of ${\mathrm{\Sigma }}_n$ on n-tuples as

$$\left[\begin{array}{c}{x}_1\\ ⋮\\ x_n\end{array}\right]{·}_2\sigma =\left[\begin{array}{c}{x}_{\sigma \left(1\right)}\\ ⋮\\ x_{\sigma \left(n\right)}\end{array}\right].$$

We will also denote by ${·}_2$ the permutation action on tuples such as $(x_1,\dots ,x_n)$ to shorten the notation.

Lemma 2.

The integration scheme from Lemma 1 induces a continuous function $F:{\mathbb{R}}^n\times {[a,b]}^n⟶{\mathbb{R}}^n$. Moreover, F is equivariant with respect to the actions defined in Definition 1.

Proof. 

We set $F^i(z_1,\dots ,z_n,x_1,\dots ,x_n):={\sum }_{k=1}^{n}G(z_k,x_i,t_k)w_k$ for each $i=1,\dots ,n$, where the points $t_1,\dots ,t_n$ are determined by the integration scheme defined in Lemma 1. We can therefore define F as the column vector ${\left[F^i\right]}_{i=1}^n$ and as a function of the tuple $(z_1,\dots ,z_n,x_1,\dots ,x_n)$. Since each function $F^i$ is continuous with respect to its entries $z_k$ and $x_i$, it follows that the function F is also continuous.

To show that the function F just defined is permutation-equivariant, using the same notations introduced above, we compute

$$\begin{array}{ccc}\hfill F\left((z_1,\dots ,z_n,x_1,\dots ,x_n){·}_1\sigma \right)& =& F(z_{\sigma \left(1\right)},\dots ,z_{\sigma \left(n\right)},x_{\sigma \left(1\right)},\dots ,x_{\sigma \left(n\right)})\hfill \\ & =& \left[\begin{array}{c}{F}^{\sigma \left(1\right)}(z_{\sigma \left(1\right)},\dots ,z_{\sigma \left(n\right)},x_1,\dots ,x_n)\\ ⋮\\ F^{\sigma \left(n\right)}(z_{\sigma \left(1\right)},\dots ,z_{\sigma \left(n\right)},x_1,\dots ,x_n)\end{array}\right]\hfill \\ & =& \left[\begin{array}{c}{\displaystyle {\sum }_{k=1}^n}G(z_{\sigma \left(k\right)},x_{\sigma \left(1\right)},t_{\sigma \left(k\right)})w_{\sigma \left(k\right)}\\ ⋮\\ {\displaystyle {\sum }_{k=1}^n}G(z_{\sigma \left(k\right)},x_{\sigma \left(n\right)},t_{\sigma \left(k\right)})w_{\sigma \left(k\right)}\end{array}\right]\hfill \\ & =& \left[\begin{array}{c}{\displaystyle {\sum }_{k=1}^n}G(z_k,x_{\sigma \left(1\right)},t_k)w_k\\ ⋮\\ {\displaystyle {\sum }_{k=1}^n}G(z_k,x_{\sigma \left(n\right)},t_k)w_k\end{array}\right]\hfill \\ & =& \left[\begin{array}{c}{F}^{\sigma \left(1\right)}(z_1,\dots ,z_n,x_1,\dots ,x_n)\\ ⋮\\ F^{\sigma \left(n\right)}(z_1,\dots ,z_n,x_1,\dots ,x_n)\end{array}\right]\hfill \\ & =& F(z_1,\dots ,z_n,x_1,\dots ,x_n){·}_2\sigma .\hfill \end{array}$$

This shows that F is equivariant with respect to the actions. □

In fact, we will only need a subcase of the previous lemma. We formulate this in the following result.

Lemma 3.

For any choice of function y in the Hölder space $C^{k,\alpha }\left([a,b]\right)$, the integration scheme from Lemma 1 induces a permutation-equivariant continuous function $F_y:{[a,b]}^n⟶{\mathbb{R}}^n$.

Proof. 

For a fixed $y\in C^{k,\alpha }\left([a,b]\right)$, the function F constructed in Lemma 2 will have the first n entries only depending on $t_i$, since they are obtained by evaluating y at the points $t_i$, for $i=1,\dots ,n$. The points $t_i$, however, are determined by the integration scheme and are therefore fixed. This induces a function $F_y:{[a,b]}^n⟶{\mathbb{R}}^n$, which is continuous. To show that the map $F_y$ is permutation-equivariant with respect to the ${·}_2$ action, we use the equivariance of F proved in Lemma 2 in the following way. First, observe that F is independent of any permutation with respect to the first n coordinates, since they are all summed together. The diagonal permutation action $\sigma$ on the entries of $F(y_1,\dots ,y_n,x_1,\dots ,x_n)$ is the same as the action of $\sigma$ on the entries of $F_y(x_1,\dots ,x_n)$ for $y_1,\dots ,y_n$, fixed as $y_r=y\left(t_r\right)$. Therefore, the equivariance of Lemma 2 completes the proof. □

Recall the following notation conventions, which will be used in the subsequent results. For a function $f:{\mathbb{R}}^n⟶{\mathbb{R}}^m$, we indicate by $f^i(x_1,\dots ,x_n)$ the $i^{\mathrm{th}}$ entry of the vector $f(x_1,\dots ,x_n)$. Following Lemma 2, we also define the function F, whose $i^{\mathrm{th}}$ component is given by $F^i(z_1,\dots ,z_n,x_1,\dots ,x_n):={\sum }_{k=1}^{n}G(z_k,x_i,t_k)w_k$ for each $i=1,\dots ,n$, and $F_y$ is induced from F as in Lemma 3. We notice that $F_y$ gives an approximation of $T^i\left(y\right)$ by construction.

Lemma 4.

Let y be in the ball ${\overline{\mathbb{B}}}_{\rho }$ of radius ρ in the Hölder space $C^{k,\alpha }\left([a,b]\right)$. Then, for any choice of $ϵ>0$, we can find a transformer $W_y:{\mathbb{R}}^n⟶{\mathbb{R}}^n$ such that

$$\begin{array}{c}\hfill {\left({\displaystyle \sum _{i=1}^n}{\int }_{[a,t_1]\times \dots \times [t_{n-1},b]}{|F_y^i(x_1,\dots ,x_n)-W_y^i(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}<ϵ.\end{array}$$

(7)

Over a compact $\mathbb{K}\subset C^{k,\alpha }\left([a,b]\right)$, this approximation can be conducted uniformly in y with a generalized transformer $\mathcal{W}$, as follows:

$$\begin{array}{c}\hfill {\left({\displaystyle \sum _{i=1}^n}{\int }_{[a,t_1]\times \cdots \times [t_{n-1},b]}{|F_y^i(x_1,\dots ,x_n)-{\mathcal{W}}^i\left(y\right)(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}<ϵ.\end{array}$$

(8)

Proof. 

Applying Lemma 3, for any choice of $y\in {\overline{\mathbb{B}}}_{\rho }$, $\mathcal{T}$ induces a continuous and permutation-equivariant map $F_y:{[a,b]}^n⟶{\mathbb{R}}^n$. Applying Theorem 2 of [16] to the permutation-equivariant map $F_y$, we can find a transformer architecture $W_y$, which depends on the given y, such that ${\parallel F_y-W_y\parallel }_p<ϵ$. Then, we have the following:

$$\begin{array}{ccc}\hfill \parallel F_y-W_y{\parallel }_p& =& {\left({\displaystyle \sum _{i=1}^n}{\int }_{{[a,b]}^n}{|F_y^i(x_1,\dots ,x_n)-W_y^i(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}\hfill \\ & \ge & {\left({\displaystyle \sum _{i=1}^n}{\int }_{[a,t_1]\times \dots \times [t_{n-1},b]}{|F_y^i(x_1,\dots ,x_n)-W_y^i(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}.\hfill \end{array}$$

We therefore obtain

$$\begin{array}{c}\hfill {\left({\displaystyle \sum _{i=1}^n}{\int }_{[a,t_1]\times \dots \times [t_{n-1},b]}{|F_y^i(x_1,\dots ,x_n)-W_y^i(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}<ϵ.\end{array}$$

For the second part of the statement, suppose that $\mathbb{K}$ is compact. Then, by the boundedness of the compacts, there exists a ball ${\mathbb{B}}_{\rho }$ of radius $\rho >0$ that contains $\mathbb{K}$. For any choice of $y\in \mathbb{K}\subset {\overline{\mathbb{B}}}_{\rho }$, we know from the first part of the lemma that $\mathcal{T}\left(y\right)$ is approximated arbitrarily well by some transformer $W_y$.

Observe that $\mathcal{T}\left(y\right)$ depends continuously on y by construction (see Lemma 1). Moreover, as discussed in Lemma 1, in ${\overline{\mathbb{B}}}_{\rho }$, the ranges of the functions y are contained in $[-\rho ,\rho ]$. Using the Heine–Cantor Theorem for G over the interval $[-\rho ,\rho ]$ and the definition of $F_y$ (given in Lemma 2) induced by $\mathcal{T}\left(y\right)$, we can see that upon choosing an $r>0$ value that is small enough for any $y,z$ such that ${\parallel y-z\parallel }_{k,\alpha }<r$, we have the inequality

$$\begin{array}{c}\hfill {\left(\sum _i{\int }_{[a,t_1]\times \cdots \times [t_{n-1},b]}{|F_y^i(x_1\dots ,x_n)-F_z^i(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}<{\displaystyle \frac{ϵ}{2}}.\end{array}$$

(9)

We choose an r-net for the compact $\mathbb{K}$ consisting of finitely many points $y_1,\dots ,y_n$, with corresponding balls ${\mathbb{B}}_j$ with radii less than r. For each $y_j$, we find a transformer $W_j$ such that

$$\begin{array}{c}\hfill {\left({\displaystyle \sum _{i=1}^n}{\int }_{[a,t_1]\times \cdots \times [t_{n-1},b]}{|F_j^i(x_1,\dots ,x_n)-W_j^i(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}<{\displaystyle \frac{ϵ}{2}},\end{array}$$

where we use j in the subscript of $F^i$ instead of $y_j$ for simplicity. There exists a partition of unity $\left\{{\eta }_k\right\}$ subordinate to the covering $\left\{{\mathbb{B}}_k\right\}$. We use such partition of unity to define a transformer $\mathcal{W}$ as a combination of the $W_k$ as

$$\begin{array}{c}\hfill \mathcal{W}\left(y\right)=\sum _k{\eta }_k\left(y\right)W_k.\end{array}$$

For any y in $\mathbb{K}$, making use of Jensen’s inequality, we have

$$\begin{array}{cc}& {\displaystyle {\displaystyle \sum _{i=1}^n}{\int }_{\Omega }|F_y^i(x_1,\dots ,x_n)-{\mathcal{W}}^i\left(y\right)(x_1,\dots ,x_n){|}^{p}d\mu }\hfill \\ & \hspace{1em}={\displaystyle \sum _{i=1}^n}{\int }_{\Omega }|\sum _k{\eta }_k\left(y\right)F_y^i(x_1,\dots ,x_n)-\sum _k{\eta }_k\left(y\right)W_k^i(x_1,\dots ,x_n){|}^{p}d\mu \hfill \\ & \hspace{1em}\le {\displaystyle \sum _{i=1}^n}\sum _k{\eta }_k\left(y\right){\int }_{\Omega }|F_y^i(x_1,\dots ,x_n)-W_k^i(x_1,\dots ,x_n){|}^{p}d\mu \hfill \\ & \hspace{1em}\le {\displaystyle \sum _{i=1}^n}\left[2^{p-1}\sum _k{\eta }_k\left(y\right){\int }_{\Omega }\right|F_y^i(x_1,\dots ,x_n)-F_k^i(x_1,\dots ,x_n){|}^{p}d\mu \hfill \\ & \hspace{1em}\hspace{1em}+2^{p-1}\sum _k{\eta }_k\left(y\right){\int }_{\Omega }|F_k^i(x_1,\dots ,x_n)-W_k^i(x_1,\dots ,x_n){|}^{p}d\mu ]\hfill \\ & \hspace{1em}=2^{p-1}\sum _k{\eta }_k\left(y\right){\displaystyle \sum _{i=1}^n}{\int }_{\Omega }|F_y^i(x_1,\dots ,x_n)-F_k^i(x_1,\dots ,x_n){|}^{p}d\mu \hfill \\ & \hspace{1em}\hspace{1em}+2^{p-1}\sum _k{\eta }_k\left(y\right){\displaystyle \sum _{i=1}^n}{\int }_{\Omega }|F_k^i(x_1,\dots ,x_n)-W_k^i(x_1,\dots ,x_n){|}^{p}d\mu \hfill \\ & \hspace{1em}<2^{p-1}\sum _k{\eta }_k\left(y\right){\displaystyle \frac{{ϵ}^p}{2^p}}+2^{p-1}\sum _k{\eta }_k\left(y\right){\displaystyle \frac{{ϵ}^p}{2^p}}\hfill \\ & \hspace{1em}={\displaystyle \frac{{ϵ}^p}{2}}+{\displaystyle \frac{{ϵ}^p}{2}}\hfill \\ & \hspace{1em}={ϵ}^p,\hfill \end{array}$$

where $\Omega =[a,t_1]\times \dots \times [t_{n-1},b]$. Taking p-roots, we complete the proof. □

We consider the same class of maps as in the discussion preceding Lemma 1. Namely, we let $T:X⟶X$ denote an Urysohn integral operator, such that the kernel of T is a function $G:\mathbb{R}\times [a,b]\times [a,b]⟶\mathbb{R}$ that is continuous with respect to all variables and is continuously differentiable with respect to the first and third variables. We are now in a position to prove the main result of this article.

Theorem 1.

Let $T:C^{k,\alpha }\left([a,b]\right)⟶C^{k,\alpha }\left([a,b]\right)$ be an integral operator as above, and let $\mathbb{K}\subset C^{k,\alpha }\left([a,b]\right)$ be a compact set. Then, for any $ϵ>0$ and every choice of $p\ge 1$, there exists a generalized transformer architecture $\mathcal{W}$ and a discretization of $[a,b]$ (which depend on $\mathbb{K}$, ϵ and p) such that

$$\begin{array}{c}\hfill {\left(\sum _i{\int }_{\Omega }|T^i\left(y\right)(x_1,\dots ,x_n)-{\mathcal{W}}^i\left(y\right)(x_1,\dots ,x_n){|}^p\right)}^{{\displaystyle \frac{1}{p}}}<ϵ,\end{array}$$

for every $y\in \mathbb{K}$, where $\Omega =[a,t_1]\times \cdots \times [t_{n-1},b]$ is determined by the discretization of $[a,b]$.

Proof. 

Without loss of generality, we set $[a,b]=[0,1]$. We let ${\overline{B}}_{\rho }$ denote a ball containing $\mathbb{K}$. Applying Lemma 1, we find an integration scheme $\mathcal{T}$ such that

$$\begin{array}{c}\hfill {\parallel T\left(y\right)-\mathcal{T}\left(y\right)\parallel }_{\infty }<{\displaystyle \frac{ϵ}{2}},\end{array}$$

with some discretization $0=t_1,\dots ,t_n=1$. We can assume that the discretization is uniform, with subintervals of length ${\displaystyle \frac{1}{n}}$, since we can otherwise pass to a finer partition with this property, with $\mathcal{T}$ still satisfying the previous estimate. Using Lemma 3, we find that $\mathcal{T}$ induces a continuous permutation-equivariant map, which we still denote by $\mathcal{T}$ by an abuse of notation. We apply Lemma 4 as follows. Over the compact $\mathbb{K}$, there exists a generalized transformer $\mathcal{W}$ such that

$${\left({\displaystyle \sum _{i=1}^n}{\int }_{\Omega }{|{\mathcal{T}}^i\left(y\right)(x_1,\dots ,x_n)-{\mathcal{W}}^i\left(y\right)(x_1,\dots ,x_n)|}^{p}d\mu \right)}^{{\displaystyle \frac{1}{p}}}<{\displaystyle \frac{ϵ}{2}},$$

where we have set again for simplicity $\Omega =[0,t_1]\times \cdots \times [t_{n-1},1]$ to shorten notation, where $t_i$ are the points of the aforementioned discretization.

Therefore, we have

$$\begin{array}{cc}& {\displaystyle \sum _i{\int }_{\Omega }{|T^i\left(y\right)(x_1,\dots ,x_n)-{\mathcal{W}}^i\left(y\right)(x_1,\dots ,x_n)|}^{p}d\mu }\hfill \\ & \hspace{1em}\le {\displaystyle \sum _{i=1}^n}{2}^{p-1}{\int }_{\Omega }{|T^i\left(y\right)(x_1,\dots ,x_n)-{\mathcal{T}}^i\left(y\right)(x_1,\dots ,x_n)|}^{p}d\mu \hfill \\ & \hspace{1em}\hspace{1em}+{\displaystyle \sum _{i=1}^n}{2}^{p-1}{\int }_{\Omega }{|{\mathcal{T}}^i\left(y\right)(x_1,\dots ,x_n)-{\mathcal{W}}^i\left(y\right)(x_1,\dots ,x_n)|}^{p}d\mu \hfill \\ & \hspace{1em}<{\displaystyle \sum _{i=1}^n}(2^{p-1}{\displaystyle \frac{{ϵ}^p}{2^p{n}^n}}+2^{p-1}{\displaystyle \frac{{ϵ}^p}{2^p{n}^n}})\hfill \\ & \hspace{1em}\le {\displaystyle \frac{{ϵ}^p}{2}}+{\displaystyle \frac{{ϵ}^p}{2}}\hfill \\ & \hspace{1em}={ϵ}^p\hfill \end{array}$$

for all y in $\mathbb{K}$. Taking p-roots completes the proof. □

Remark 1.

We note that no special property in the proofs above is used regarding the architecture of the transformer, except the fact that transformers can approximate equivariant sequence-to-sequence maps in the norm (1) with arbitrary accuracy. As a consequence, the approach of this article can be used to show that any universal approximator of equivariant sequence-to-sequence mappings can be used to approximate integral operators in Hölder space with arbitrary precision. Our main focus on transformers only stems from the fact that they are widely used in the study of operator learning problems.

Remark 2.

The approximation result of Theorem 1 refers to a p-norm that is an average over the points where the tokens for the transformer can be sampled. While the error “on average” is small, for the learned operator, this does not mean that certain choices of values cannot incur a significant spike in the error. This type of behavior has in fact been observed concretely in [31], where such error spikes were avoided by training the transformer model in a Sobolev space that functioned for regularization.

The construction of the partition of unity over a compact subset of a normed space can be achieved via the same procedure that produces Leray–Schauder mappings, and it is therefore implementable numerically in practice. In fact, by fixing $ϵ>0$, we can cover a compact via finitely many balls centered at functions $f_i$. With ${\mu }_i$ defined as in (2), setting ${\eta }_i\left(x\right)={\displaystyle \frac{{\mu }_i\left(x\right)}{{\sum }_i{\mu }_j\left(x\right)}}$ then gives a partition of unity subordinate to the initial covering. The functions $f_i$ used to cover the dataset space can be learned by approximating them via neural networks.

Example 1.

Here, we provide a discussion regarding possible applications of Theorem 1, along with some classes of problems that can be treated with these methods. A detailed study related to these considerations would be desirable.

Let $\mathbb{K}$ be a compact subset of $C^{k,\alpha }\left([a,b]\right)$. The metric entropy, or Kolmogorov ε-entropy, measures the number of balls of radius $\epsilon >0$ needed to cover $\mathbb{K}$. Therefore, we can estimate the number of functions in the partitions of unity used in the proofs based on the metric properties of $\mathbb{K}$ and the chosen accuracy. Once the partitions of unity have been fixed, we can approximate the target operator by a combination of paritions of unity and sequence-to-sequence maps through the discretization of the integration space, as in the proof of Theorem 1 and its preliminary lemmas. The results and methods of [17] give some asymptotic estimates on the parameters needed to approximate sequence-to-sequence functions, and therefore, combining the two approaches, it is possible to attempt an a priori study of the size of the architectures and number of transformers needed to approximate a target operator.

Classes of examples of problems that are studied via integral operators of the Urysohn type with sufficiently smooth kernels are nonlocal problems such as, for example, the Amari [32,33] and Wilson–Cowan [34] models in computational neuroscience. These models give rise to continuous maps between Hölder spaces [35] due to the regularity of the Urysohn kernels. Therefore, it is in principle possible to estimate, via the procedure described above, the size of the transformer architectures needed to approximate the operators on given compact subsets of $C^{k,\alpha }\left([a,b]\right)$. More generally, we expect that a similar approach can be applied to problems described via Volterra and Fredholm integral equations, or PDEs with sufficiently regular nonlocal interactions.

While this is not the main scope of this article, we provide another universal approximation result for transformer architectures that implement a Leray–Schauder mapping, similar to [36]. For this purpose, we define a Leray–Schauder transformer to consist of a Leray–Schauder mapping composed of either a traditional transformer architecture with positional encoding as in [16,23] or the formulation of attention found in [15]. Let P and $P^{\prime }$ denote two Leray–Schauder mappings $P:U⟶X$ and $P^{\prime }:U^{\prime }⟶Y$, where U and $U^{\prime }$ are open sets containing two given compacts $\mathbb{K}$ and ${\mathbb{K}}^{\prime }$. Then, $P\left(\mathbb{K}\right)$ and $P^{\prime }\left({\mathbb{K}}^{\prime }\right)$ both the span finite dimensional subspaces E and $E^{\prime }$ of X and Y, respectively, by construction. We let $\phi$ and ${\phi }^{\prime }$ denote two linear homeomorphisms $\phi :E⟶{\mathbb{R}}^n$ and ${\phi }^{\prime }:E^{\prime }⟶{\mathbb{R}}^m$, where we set $dimE=n$ and $dim{E}^{\prime }=m$. Let us indicate by W a transformer architecture as in [15] or [16,23], where $W:{\mathbb{R}}^n⟶{\mathbb{R}}^m$. Then, a Leray–Schauder transformer $\mathfrak{T}$ is defined by the composition ${\phi }^{\prime -1}W\phi P:U\subset X⟶E^{\prime }\subset Y$.

Then, we have the following result.

Theorem 2.

Let $T:X⟶Y$ be an operator between Banach spaces, and let $\mathbb{K}\subset X$ be a compact set. Then, by fixing $ϵ>0$ arbitrarily, there exists a Leray–Schauder transformer $\mathfrak{T}$ that approximates T with precision ϵ on $\mathbb{K}$, as follows:

$$\begin{array}{c}\hfill {\parallel T\left(x\right)-\mathfrak{T}\left(x\right)\parallel }_Y<ϵ,\end{array}$$

for all $x\in \mathbb{K}$.

Proof. 

The proof consists of an application of the results in [15,37]. As shown in Theorem 2.2 in [37], we can assume without loss of generality that T is uniformly continuous on a neighborhood V of $\mathbb{K}$, upon possibly replacing it by an approximation of T. Then, we can proceed as in the proof of Theorem 2.1 in [37] as follows. Since $\mathbb{K}$ is compact, we can construct a Leray–Schauder mapping $P:U⟶X$ defined and continuous in an open neighborhood $U\subseteq V$ of $\mathbb{K}$, such that ${\parallel x-P\left(x\right)\parallel }_X<\delta$, where $\delta >0$ is chosen in such a way that $\parallel T\left(z_1\right)-T\left(z_2\right){\parallel }_Y<{\displaystyle \frac{ϵ}{3}}$ whenever $\parallel z_1-z_2{\parallel }_X<\delta$. From the construction of P, one sees also that $P\left(\mathbb{K}\right)\subset U$. Moreover, the image of P spans a finite dimensional subspace E of X, which is linearly homeomorphic to ${\mathbb{R}}^n$ for some n via a linear map $\phi :E⟶{\mathbb{R}}^n$.

Similarly, we can construct another Leray–Schauder mapping $P^{\prime }$ on the compact $T\left(P\right(\mathbb{K}\left)\right)$, such that $\parallel y-P^{\prime }{\left(y\right)\parallel }_Y<{\displaystyle \frac{ϵ}{3}}$ for each $y\in T\left(P\right(\mathbb{K}\left)\right)$. Moreover, $P^{\prime }$ can be continuously extended to the whole space Y. The image $P^{\prime }\left(T\left(P\left(\mathbb{K}\right)\right)\right)$ is a finite dimensional subspace $E^{\prime }$ of Y that is linearly homeomorphic to ${\mathbb{R}}^m$ for some m via ${\phi }^{\prime }:E^{\prime }⟶{\mathbb{R}}^m$.

The map T induces a continuous mapping $T_{n,m}:{\mathbb{R}}^n⟶{\mathbb{R}}^m$ that can be approximated in the uniform norm over the compact $\phi \left(P\left(\mathbb{K}\right)\right)\subset {\mathbb{R}}^n$ by applying Corollary 3 in [15] or Theorem 1 in [23] (in conjunction with the traditional universal approximation results of functions by convolutional neural networks). In other words, we can approximate $T_{n,m}$ with arbitrary precision in the uniform norm with a polynomial-transformer architecture without positional encoding, as in [15], or with a traditional transformer architecture with positional encoding, as in [23]. Then, as in [37], we can show that the estimate ${\parallel T\left(x\right)-\mathfrak{T}\left(x\right)\parallel }_Y\le {\parallel T\left(x\right)-TP\left(x\right)\parallel }_Y+\parallel TP\left(x\right)-P^{\prime }{TP\left(x\right)\parallel }_Y+{\parallel P^{\prime }TP\left(x\right)-\mathfrak{T}\left(x\right)\parallel }_Y<{\displaystyle \frac{ϵ}{3}}+{\displaystyle \frac{ϵ}{3}}+{\displaystyle \frac{ϵ}{3}}=ϵ$, where $\mathfrak{T}={\phi }^{\prime -1}W\phi P:U\subset X⟶E^{\prime }\subset Y$.

□

We notice that the result of Theorem 2 gives a more general version of the results in [19], since their result is applicable to the case where X has the approximation property, a restriction that is not required in the present work. Here, the Leray–Schauder mappings play the role of the encoder used in [19].

To better elucidate the differences between Theorem 2 and the approximation results of [19], we provide a more detailed comparison.

(i)

The spaces X and Y in [19] are function spaces, defined over a compact subset of ${\mathbb{R}}^n$, while in our case’s X and Y can be arbitrary Banach spaces.

(ii)

While X and Y in [19] are required to possess the approximation property, this is not needed in Theorem 2.

(iii)

No decoder is used in the present work, while the approximation found in [19] uses a decoder.

(iv)

The role of the encoder in [19] is played in Theorem 2 by the the Leray–Schauder mappings, which are explicitly and analytically defined (see Section 2).

4. Universal Approximation of Operators by Neural Integral Operators

In this section, we consider the universal approximation property of operators in Banach spaces, where the approximator is constructed through a neural integral operator in a suitable generalized sense.

Theorem 3.

Let X and Y be Banach spaces, let $T:X⟶Y$ be an operator that is twice continuously Fréchet differentiable, and let $\mathbb{K}\subset X$ be a compact subset of X. For any $ϵ>0$, there exists an integral operator $U:X⟶Y$ such that

$$\begin{array}{c}\hfill {\parallel T\left(x\right)-U\left(x\right)\parallel }_Y<ϵ,\end{array}$$

(10)

for all $x\in \mathbb{K}$.

Proof. 

We use the Taylor expansion for nonlinear operators found in Chapter 17 of Kantorovich–Akilov [27], where integration is intended as a Riemann integral over a Banach space as in [26]. See also [38,39]. Recall ([26,27]) that for $R:[x_0,\overline{x}]\subset X\to B(X,Y)$, where $B(X,Y)$ denotes the space of bounded linear operators from X to Y, the integral of R is defined (when it is well defined, e.g., when R is continuous) as

$$\begin{array}{c}\hfill {\int }_{x_0}^{x_0+\Delta x}R\left(x\right)dx={\int }_0^{1}R(x_0+t\Delta x)\Delta xdt=\underset{n\to \infty }{lim}{\displaystyle \sum _{k=0}^{n-1}}R(x_0+{\tau }_k\Delta x)\Delta x(t_{k+1}-t_k),\end{array}$$

where ${\tau }_k\in [t_k,t_{k+1}]$ for all k. For a choice of $x_0\in \mathbb{K}$ and for any $\overline{x}\in X$ such that T is twice continuously Fréchet differentiable over the interval $[x_0,\overline{x}]$ (i.e., the set of vectors $t\overline{x}+(1-t)x_0$ with $t\in [0,1]$), we can write (Taylor’s Theorem at degree 2)

$$\begin{array}{ccc}\hfill T\left(\overline{x}\right)& =& T\left(x_0\right)+T^{\prime }\left(x_0\right)(\overline{x}-x_0)+{\int }_{x_0}^{\overline{x}}{T}^{\prime \prime }\left(x\right)(\overline{x}-x,·)dx,\hfill \end{array}$$

(11)

where · is a placeholder for the entry of the bounded linear operator integrand. Since $T^{\prime \prime }$ is continuous, for each point $x\in X$, we can find a ${\delta }_x^{\prime }>0$ such that $\parallel T^{\prime \prime }\left(z\right)-T^{\prime \prime }{\left(x\right)\parallel }_{B^2(X,Y)}<ϵ$ whenever ${\parallel z-x\parallel }_X<{\delta }_x^{\prime }$, where $B^2(X,Y)$ denotes the space of bounded bilinear maps from X to Y with the norm given by $\parallel A\parallel ={sup}_{{\parallel h\parallel }_X\le 1,{\parallel k\parallel }_X\le 1}{\parallel A(h,k)\parallel }_Y$. Then, for each x, define ${\delta }_x=min\{{\delta }_x^{\prime },1\}$. Since $\mathbb{K}$ is compact, we can cover it with finitely many balls $B(x_i,{\delta }_i)$, with $i=1,\dots ,n$, where ${\delta }_i$ is a shortened notation for ${\delta }_{x_i}$. We set $A\left(t\right)(·):=(T^{\prime \prime }\left(x\left(t\right)\right)-T^{\prime \prime }\left(x_i\right))((1-t)h,·)$, with $x\left(t\right)=(1-t)h+x_i$, and we again use · to indicate where the input of the linear operator $A\left(t\right)$ for each $t\in [0,1]$ is inserted. In correspondence with the balls $B(x_i,{\delta }_i)$, we define

$$\begin{array}{ccc}\hfill T_i\left(\overline{x}\right)& =& T\left(x_i\right)+T^{\prime }\left(x_i\right)(\overline{x}-x_i)+{\int }_{x_i}^{\overline{x}}{T}^{\prime \prime }\left(x_i\right)(\overline{x}-x,·)dx.\hfill \end{array}$$

(12)

Then, for any $x\in B(x_i,{\delta }_i)$, with $h=x-x_i$ and the parameterized interval $[x_i,x]\subset X$ given by $x\left(t\right)=(1-t)h+x_i$, one has

$$\begin{array}{ccc}\hfill \parallel {\int }_0^{1}A\left(t\right)dt{\parallel }_Y& \le & {\int }_0^1\parallel A\left(t\right){\parallel }_{Y}dt\hfill \\ & \le & {\int }_0^1{(1-t)\parallel h\parallel }_X^2\parallel T^{\prime \prime }\left(x\left(t\right)\right)-T^{\prime \prime }\left(x_i\right){\parallel }_{B^2(X,Y)}dt\hfill \\ & <& ϵ.\hfill \end{array}$$

Therefore, for each $x\in B(x_i,{\delta }_i)$, we have

$$\begin{array}{ccc}\hfill \parallel T\left(x\right)-T_i{\left(x\right)\parallel }_Y& =& \parallel T\left(x_i\right)+T^{\prime }\left(x_i\right)(\overline{x}-x_i)+{\int }_{x_i}^x{T}^{\prime \prime }\left(z\right)(x-z,·)dz\hfill \\ & & -T\left(x_i\right)-T^{\prime }\left(x_i\right)(\overline{x}-x_i)-{\int }_{x_i}^x{T}^{\prime \prime }\left(x_i\right)(x-z,·)dz{\parallel }_Y\hfill \\ & =& \parallel {\int }_{x_i}^x(T^{\prime \prime }\left(z\right)-T^{\prime \prime }\left(x_i\right))(x-z,·)dz{\parallel }_Y\hfill \\ & =& \parallel {\int }_0^1(1-t)\left(T^{\prime \prime }((1-t)h+x_i)\right)(h,h)dt{\parallel }_Y\hfill \\ & =& \parallel {\int }_0^{1}A\left(t\right)dt{\parallel }_Y\hfill \\ & <& ϵ.\hfill \end{array}$$

The space X is Hausdorff and paracompact, and we can therefore find a partition of unity ${\left\{{\eta }_j\right\}}_{j=1}^n\cup \left\{{\eta }_{\infty }\right\}$ subordinate to the open cover ${\left\{B(x_i,{\delta }_i)\right\}}_{i=1}^n\bigcup \{X-\mathbb{K}\}$—see [40]—where we use the notation ${\eta }_{\infty }$ to denote the element of the partition with support contained in $X-\mathbb{K}$. We now define the operator $U:X⟶Y$ by patching together the operators $T_i:X⟶Y$. In fact, for any $x\in X$, we set

$$U\left(x\right)={\displaystyle \sum _{i=1}^n}{\eta }_i\left(x\right)T_i\left(x\right)+{\eta }_{\infty }\left(x\right)y_{\infty },$$

where $y_{\infty }$ is arbitrarily chosen. For any $x\in \mathbb{K}$, there exist $i_1,\dots ,i_d\ne \infty$ such that $x\in B(x_i,{\delta }_i)$ for $i=i_1,\dots ,i_d$ and ${\sum }_{i=i_1,\dots ,i_d}{\eta }_i\left(x\right)=1$, while ${\eta }_i\left(x\right)=0$ for all $i\ne i_1,\dots ,i_d$. Then, we have

$$\begin{array}{ccc}\hfill \parallel T\left(x\right)-U\left(x\right)\parallel & =& \parallel \sum _i{\eta }_i\left(x\right)T\left(x\right)-\sum _i{\eta }_i\left(x\right)T_i\left(x\right)\parallel \hfill \\ & =& \parallel \sum _{i=i_1,\dots ,i_d}{\eta }_i\left(x\right)T\left(x\right)-\sum _{i=i_1,\dots ,i_d}{\eta }_i\left(x\right)T_i\left(x\right)\parallel \hfill \\ & \le & \sum _{i=i_1,\dots ,i_d}{\eta }_i\left(x\right)\parallel T\left(x\right)-T_i\left(x\right)\parallel \hfill \\ & <& \sum _{i=i_1,\dots ,i_d}{\eta }_i\left(x\right)ϵ\hfill \\ & =& ϵ,\hfill \end{array}$$

This completes the proof. □

In the following, by the Gavuring neural integral operator $\mathfrak{T}\left(x\right)$, we mean the class of neural network architectures defined in Section 2.3.

Theorem 4.

Let $T:X⟶Y$ and $\mathbb{K}$ be as in Theorem 3. Then, for any choice of $ϵ>0$, there exists a Gavurin neural integral operator $\mathfrak{T}$ such that

$$\begin{array}{c}\hfill {\parallel T\left(x\right)-\mathfrak{T}\left(x\right)\parallel }_Y<ϵ,\end{array}$$

for any $x\in \mathbb{K}$.

Proof. 

We apply Theorem 3 to approximate T via an integral operator U, where

$$\begin{array}{c}U\left(y\right)=\sum _k{\eta }_k\left(y\right)T\left(y_k\right)+\sum _k{\eta }_k\left(y\right)T^{\prime }\left(y_k\right)(y-y_k)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hfill \\ \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\sum _k{\eta }_k\left(y\right){\int }_0^1{T}^{\prime \prime }\left(y_k\right)((1-t)(y-y_k),y-y_k)dt,\end{array}$$

in such a way that ${\parallel T\left(y\right)-U\left(y\right)\parallel }_Y<{\displaystyle \frac{ϵ}{2}}$, where $y_k$ are finitely many elements of $\mathbb{K}$, as in the proof of Theorem 3.

Let us consider the integral part of the operator U, namely

$$\begin{array}{c}\hfill A\left(y\right)=\sum _k{\eta }_k\left(y\right){\int }_0^1{T}^{\prime \prime }\left(y_k\right)((1-t)(y-y_k),y-y_k)dt=\sum _k{\eta }_k\left(y\right)A_k\left(y\right),\end{array}$$

where we set $A_k\left(y\right):={\int }_0^1{K}_k(t,y)dt$, with $K_k(t,y):=T^{\prime \prime }\left(y_k\right)((1-t)(y-y_k),y-y_k)$. We have that $K_k$ is continuous with respect to t and y, and therefore uniformly continuous on $[0,1]\times \mathbb{K}$. We select $\delta >0$ such that $\parallel K_k(t_1,y_1)-K_k(t_2,y_2){\parallel }_Y<{\displaystyle \frac{ϵ}{2}}$ whenever $|t_1-t_2|<\delta$ and $\parallel y_1-y_2{\parallel }_X<\delta$, with $t_1,t_2\in [0,1]$ and $y_1,y_2\in \mathbb{K}$. Given this choice of $\delta$, we select a partition of points $0=t_0,t_1,\dots ,t_{d-1},t_d=1$ such that $|t_i-t_{i+1}|<\delta$ for all $i=0,1,\dots ,d-1$. Using Theorem 2.1 in [37], we can find a neural network ${\widehat{K}}_{ij}$, consisting of Leray–Schauder mappings and a deep neural network, such that $\parallel K_i\left(t_j\right)\left(y\right)-{\widehat{K}}_{ij}\left(y\right){\parallel }_Y<{\displaystyle \frac{ϵ}{2}}$ for all $y\in \mathbb{K}$. Let us now consider the open covering of $[0,1]$ with $B_{\delta }\left(t_j\right)$ open balls centered at $t_j$, with $j=0,\dots ,d$, and radius $\delta$. Let ${\alpha }_k$ denote a partition of unity subordinate to $\left\{B_{\delta }\left(t_j\right)\right\}$. One can directly verify that the operator ${\widehat{K}}_i\left(t\right):={\sum }_k{\alpha }_k\left(t\right){\widehat{K}}_{ik}$ satisfies the property that $\parallel K_i\left(t\right)\left(y\right)-{\widehat{K}}_i{\left(t\right)\left(y\right)\parallel }_Y<{\displaystyle \frac{ϵ}{2}}$ whenever y is in $\mathbb{K}$. Since this construction can be performed for each i, it follows that the integral part of the operator U can be approximated by an integral operator with a kernel consisting of Leray–Schauder mappings and (deep) neural networks.

Another application of Theorem 2.1 in [37] allows us to approximate ${\sum }_k{\eta }_k\left(y\right)T^{\prime }\left(y_k\right)(y-y_k)$ with a deep neural network (and Leray–Schauder mappings) with precision ${\displaystyle \frac{ϵ}{2}}$ over $\mathbb{K}$. Putting both approximations together completes the proof. □

5. Further Perspectives

We conclude this article with some observations for further investigation.

We would like to mention that the results of Theorem 1 approximate integral operators between Hölder spaces, where the need for Hölder norms arises from the use of numerical quadrature to approximate integrals. Therefore, one might ask under what conditions transformers are universal approximators of integral operators between $L^p$ spaces. This question is yet to be answered.

Theorem 2 gives an approximation result for transformers in complete generality—note that it refers to general Banach spaces, rather than some specific Banach space of functions. Once a norm is fixed, the Leray–Schauder mapping has an explicit form that is analytical, and therefore, it seems that the new architecture is completely transparent. In general, however, it is useful to learn the Leray–Schauder mappings as well to increase the applicability of the model. This is done, for example, in [36], where the resulting model is tested (in the $L^p$-space) on some integral equations and PDEs. The theoretical guarantees are also studied. It would be interesting to investigate extensions of these results in relation to the work in this article. The main difficulty in extending the present work to the case of $L^p$-spaces, for example, is that there is no bound on the derivatives that can control the error associated to an integration scheme. Therefore, an analogous version of Lemma 1 would fail. It is possible to attempt similar approaches in the Sobolev spaces, as cubature formulas can control the integration errors. However, we will leave this interesting line of work to future research.

Lastly, the results of Section 4 show that mappings between general Banach spaces can be approximated through a generalized version of the neural integral operator based on the Gavurin integral. While this is an interesting theoretical development, it is still unclear whether removing the use of the Gavurin integral would still allow for such a general approximation result.