The Theory and Applications of Hölder Widths

Abstract

We introduce the Hölder width, which measures the best error performance of some recent nonlinear approximation methods, such as deep neural network approximation. Then, we investigate the relationship between Hölder widths and other widths, showing that some Hölder widths are essentially smaller than $n$-Kolmogorov widths and linear widths. We also prove that, as the Hölder constants grow with n, the Hölder widths are much smaller than the entropy numbers. The fact that Hölder widths are smaller than the known widths implies that the nonlinear approximation represented by deep neural networks can provide a better approximation order than other existing approximation methods, such as adaptive finite elements and $n$-term wavelet approximation. In particular, we show that Hölder widths for Sobolev and Besov classes, induced by deep neural networks, are $\mathcal{O}\left(n^{-2s/d}\right)$ and are much smaller than other known widths and entropy numbers, which are $\mathcal{O}\left(n^{-s/d}\right)$.

1. Introduction

Width theory is one of the most important topics in approximation theory because widths can be considered approximation standards that indicate the accuracy achievable for a given function class using an approximation method. They have been extensively studied and applied in various fields, providing a benchmark for the best performance of different approximation techniques. One of the earliest studies on widths was Kolmogorov’s work in 1936, where he introduced the concept of $n$-Kolmogorov widths [1]. With the development of modern science and engineering, the theory of widths has also developed rapidly, greatly promoting research into various linear and nonlinear approximation methods. Problems related to width theory have been and continue to be studied by many experts, including Pinkus, Lorentz, DeVore, Temlyakov, et al. [2,3,4,5,6,7,8,9,10,11,12]. In addition, nonlinear methods play a crucial role in understanding complex phenomena across various applications, such as compressed sensing, signal processing, and neural networks [13,14,15]. Widths such as manifold widths, nonlinear widths, and Lipschitz widths have been utilized as fundamental measures to assess the optimal convergence rate of these nonlinear methods [12,15,16,17,18].

It is known that neural networks can serve as powerful nonlinear tools. For instance, the ReLU (Rectified Linear Unit) activation function,

$$\sigma \left(t\right):=\mathrm{ReLU}\left(t\right)=max\{0,t\},t\in \mathbb{R},$$

is characterized as a Lipschitz mapping, which has led to the introduction of stable manifold widths and Lipschitz widths. In [13,17], Cohen et al. and DeVore, et al. investigated stable manifold widths to quantify error performance in nonlinear approximation methods, such as compressed sensing and neural networks. They discussed the fundamental properties of these widths and established their connections with entropy numbers. In [18,19], Petrova and Wojtaszczyk introduced Lipschitz widths and showed their relationships with other widths and entropy numbers. However, not all mappings are Lipschitz; thus, it is essential to consider weaker conditions to understand the error performance of nonlinear approximation methods. One such condition is the Hölder mapping, which we explore in this paper. We will introduce the concept of Hölder widths and investigate their relationship with other widths and entropy numbers. Our results may provide a better understanding of the effects of such nonlinear approximation methods and their potential applications in deep neural networks.

Many authors have achieved profound results for the ReLU activation function, which acts as a $Lip1$ continuous function in feed-forward deep neural networks (DNNs) [14,17,18,20,21]. It is known that the mapping $\mathsf{\Phi }:(B_{l_{\infty }^{cn}}{,\parallel ·\parallel }_{l_{\infty }^{cn}})\to C\left({[0,1]}^d\right)$ in [18] with the ReLU activation function is a $C^{\prime }n{W}^n$ Lipschitz mapping in DNNs, where the network has a width W and adepth n. Unlike the Lipschitz width, the terms ‘width’ and ‘depth’ here indicate the scale of the network. The performance of this Lipschitz mapping is discussed in [14].

We introduce a more flexible assumption. Let $(Y,d)$ and $(Z,\rho )$ be metric spaces. Moreover, we assume that the space Z is separable. Define $\mathsf{\Phi }:(Y,d)\to (Z,\rho )$ as an $\alpha$-Hölder mapping with coefficient $\gamma$ if for any $x,y\in Y$,

$$\rho \left(\mathsf{\Phi }\left(x\right),\mathsf{\Phi }\left(y\right)\right)\le \gamma d^{\alpha }\left(x,y\right),\gamma >0\mathrm{and}0<\alpha \le 1.$$

We could also say that $\mathsf{\Phi }$ satisfies the $H_{\alpha }\left(\gamma \right)$ condition [22] equivalent to

$$\underset{x,y\in Y}{sup}{\displaystyle \frac{\rho \left(\mathsf{\Phi }\left(x\right),\mathsf{\Phi }\left(y\right)\right)}{d^{\alpha }\left(x,y\right)}}\le \gamma ,\gamma >0\mathrm{and}0<\alpha \le 1.$$

We provide some remarks on Hölder mappings below.

Remark 1.

If $\alpha =1$, then Φ is Lipschitz continuous.

Remark 2.

If Y is bounded and Φ is an α-Hölder mapping, then for any $\beta \le \alpha$, Φ is a β-Hölder mapping.

Remark 3.

The minimum α-Hölder coefficient $\gamma =0$ if and only if Φ is constant.

Note that the RePU (Rectified Power Unit) activation function [23,24],

$${\sigma }_1\left(t\right):=\mathrm{RePU}\left(t\right)=max{\{0,t\}}^{\alpha },t\in \mathbb{R},$$

with $\alpha \in \mathbb{N},\alpha \ge 2$, and the GELU (Gaussian Error Linear Unit) activation function [25],

$${\sigma }_2\left(t\right):=\mathrm{GELU}\left(t\right)=0.5t\left(1+tanh\left(\sqrt{{\displaystyle \frac{2}{\pi }}}(t+0.044715t^3)\right)\right),t\in \mathbb{R}.$$

can be considered 1-Hölder mappings in bounded spaces. The performance of these mappings can be found in [14,18]. Moreover, there are various $\alpha$-Hölder activation functions with $0<\alpha <1$. In [26], Forti, Grazzini, et al. obtained global convergence results, where the neuron activations were modeled by ${\alpha }_i$-Hölder continuous functions with ${\alpha }_i\in$ (0, 1), such as

$${\sigma }_i\left(t\right)=k_i\mathrm{sign}\left(t\right){\left|t\right|}^{{\alpha }_i},$$

where $k_i>0$ and $i\in {\theta }_C$ defined in [26]. These activations can significantly increase the computational power [26,27,28]. Motivated by the above results, we mainly focus on the $\alpha$-Hölder condition with $0<\alpha <1$, which is weaker than the Lipschitz condition.

Now, we introduce Hölder widths, which measure the best error performance of some recent nonlinear approximation methods characterized by Hölder mappings. Throughout this paper, let X be a Banach space with a norm ${\parallel ·\parallel }_X$, and let $Y_n$ be an $n$-dimensional Banach space with a norm ${\parallel ·\parallel }_{Y_n}$ on ${\mathbb{R}}^n$, $n\ge 1$. Denote the unit ball of $Y_n$ by $B\left(Y_n\right):=\{y\in {\mathbb{R}}^n{:\parallel y\parallel }_{Y_n}\le 1\}$.

Let K be a bounded subset of X. For $\gamma \ge 0$, $0<\alpha <1$, we define the fixed Hölder widths

$${\delta }_n^{\gamma ,\alpha }{(K,Y_n)}_X:=\underset{{\mathsf{\Phi }}_n}{inf}\underset{f\in K}{sup}\underset{y\in B\left(Y_n\right)}{inf}{\parallel f-{\mathsf{\Phi }}_n\left(y\right)\parallel }_X,$$

(1)

where ${\mathsf{\Phi }}_n:\left(B\left(Y_n\right),{\parallel ·\parallel }_{Y_n}\right)\to X$ satisfies

$$\underset{x,y\in B\left(Y_n\right)}{sup}{\displaystyle \frac{{\parallel \mathsf{\Phi }\left(x\right)-\mathsf{\Phi }\left(y\right)\parallel }_X}{{\parallel x-y\parallel }_{Y_n}^{\alpha }}}\le \gamma .$$

Next, we define the Hölder width

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X:=\underset{{\parallel ·\parallel }_{Y_n}}{inf}{\delta }_n^{\gamma ,\alpha }{(K,Y_n)}_X,$$

(2)

where the infimum is taken over all norms ${\parallel ·\parallel }_{Y_n}$ on ${\mathbb{R}}^n$. From definition (1), we see that the error of any numerical method based on Hölder mappings will not be smaller than the Hölder widths.

We propose the Hölder widths, exploring their properties and relationships with other known widths and entropy numbers. In Section 2, we establish the fundamental properties of Hölder widths. In Section 3, we compare Hölder widths with $n$-Kolmogorov widths, linear widths, and nonlinear $(n,N)$-widths. In Section 4, we investigate the relationship between Hölder widths and entropy numbers. In Section 5, we provide some specific applications and derive the asymptotic order of Hölder widths for Sobolev classes $B{W}_p^s\left({[0,1]}^d\right)$ and Besov classes $B{B}_{p,\tau }^s\left({[0,1]}^d\right)$ using deep neural networks. In Section 6, we provide some concluding remarks. All detailed proofs for the results from Section 2, Section 3 and Section 4 are included in Appendix A, Appendix B and Appendix C, and the proofs for Theorems 10 and 15 in Section 5 are provided in Appendix D.

2. Fundamental Properties of Hölder Widths

Recall that the radius of a set $K\subset X$ is defined as

$$\mathrm{rad}\left(K\right)=:\underset{g\in X}{inf}\underset{f\in K}{sup}{\parallel f-g\parallel }_X.$$

(3)

It is known from Remark 3 that a function that satisfies the $H_{\alpha }\left(0\right)$ condition is a constant function. Then, for the $n$-dimensional space $Y_n$,

$$\mathrm{rad}\left(K\right)={\delta }_n^{0,\alpha }{(K,Y_n)}_X={\delta }_n^{0,\alpha }{\left(K\right)}_X.$$

(4)

Moreover, for a fixed constant $\alpha$, it is known from (2) that ${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X$ is decreasing with respect to $\gamma$ and n, that is, (i) if ${\gamma }_1\le {\gamma }_2$, then ${\delta }_n^{{\gamma }_2,\alpha }{\left(K\right)}_X\le {\delta }_n^{{\gamma }_1,\alpha }{\left(K\right)}_X\le \mathrm{rad}\left(K\right)<\infty$, and (ii) if $n_1\le n_2$, then ${\delta }_{n_2}^{\gamma ,\alpha }{\left(K\right)}_X\le {\delta }_{n_1}^{\gamma ,\alpha }{\left(K\right)}_X$.

In addition, it is easy to see that the space (${\mathbb{R}}^n$, ${\parallel ·\parallel }_{Y_n}$) in (1) and (2) can be replaced with any $n$-dimensional normed space ($Z_n$,${\parallel ·\parallel }_{Z_n}$) such that

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X=\underset{{\parallel ·\parallel }_{Z_n}}{inf}{\delta }_n^{\gamma ,\alpha }{(K,Z_n)}_X,{\delta }_n^{\gamma ,\alpha }{(K,Z_n)}_X=\underset{{\mathsf{\Phi }}_n}{inf}\underset{f\in K}{sup}\underset{z\in B\left(Z_n\right)}{inf}{\parallel f-{\mathsf{\Phi }}_n\left(z\right)\parallel }_X,$$

where $B\left(Z_n\right):=\{z\in Z_n{:\parallel z\parallel }_{Z_n}\le 1\}.$

Denote by

$${\ell }_p^n:=({\mathbb{R}}^n{,\parallel ·\parallel }_{{\ell }_p})$$

the space ${\mathbb{R}}^n$ equipped with the ${\ell }_p$ norm, that is, for $y=(y_1,y_2,\cdots ,y_n)\in {\mathbb{R}}^n$,

$${\parallel y\parallel }_{{\ell }_{\infty }^n}:=\underset{j}{max}|y_j{|,\parallel y\parallel }_{{\ell }_p^n}:={\left(\sum _{j=1}^n{\left|y_j\right|}^p\right)}^{1/p}.$$

Recall that an $\epsilon$-covering of K is a collection $\{g_1,\cdots ,g_m\}\subset X$ such that

$$K\subset \bigcup _{j=1}^{m}B(g_j,\epsilon ).$$

The minimal $\epsilon$-covering number $N_{\epsilon }\left(K\right)$ is the minimal cardinality of the $\epsilon$-covering of K. We say that a set K is totally bounded if for every $\epsilon >0$, $N_{\epsilon }\left(K\right)<\infty$.

We establish the following fundamental properties of Hölder widths.

Theorem 1.

Let K be a compact subset of X. For any $n\in \mathbb{N}$, $\gamma >0$, and $0<\alpha <1$, there exists a norm ${\parallel ·\parallel }_Y$ on ${\mathbb{R}}^n$ satisfying for $y\in {\mathbb{R}}^n$,

$${\parallel y\parallel }_{{\ell }_{\infty }^n}\le {\parallel y\parallel }_Y\le {\parallel y\parallel }_{{\ell }_1^n},$$

(5)

such that

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X={\delta }_n^{\gamma ,\alpha }{(K,Y)}_X.$$

Theorem 2.

Let K be a compact subset of X. As a function of γ, the Hölder width ${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X$ is continuous.

Theorem 3.

A subset K of X is totally bounded if and only if for every $n\in \mathbb{N}$,

$$\underset{\gamma \to \infty }{lim}{\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X=0.$$

Theorem 4.

Let $K\subset X$ and $0<\alpha <1$. If for every $\epsilon >0$, there exist two numbers $n\in \mathbb{N}$ and $\gamma >0$ such that ${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\le \epsilon$, then K is totally bounded.

It is important to delve deeper into Hölder widths since the Hölder condition offers a more flexible framework that can accommodate a broader range of functions, making it particularly suitable for analyzing and approximating complicated functions and datasets. In the forthcoming sections, we will gain deeper insights into the approximation performance measured by Hölder widths.

3. The Relationship Between Hölder Widths and Other Widths

In this section, we will demonstrate that Hölder widths are smaller than other known widths, such as $n$-Kolmogorov, linear, and nonlinear $(n,N)$-widths. In the following sections, we assume that K is a compact subset of the Banach space X, which is our main concern.

3.1. The Relationship Between Hölder Widths, $n$-Kolmogorov Widths, and Linear Widths

We recall the definition of the $n$-Kolmogorov width of K from [2], as follows:

$$d_0{\left(K\right)}_X:=\underset{f\in K}{sup}{\parallel f\parallel }_X,d_n{\left(K\right)}_X:=\underset{dim\left(X_n\right)=n}{inf}\underset{f\in K}{sup}\underset{g\in X_n}{inf}{\parallel f-g\parallel }_X,n\ge 1,$$

where the infimum is taken over all $n$-dimensional subspaces $X_n\subset X$.

It is known that the $n$-Kolmogorov width determines the optimal errors generated by approximating the ‘worst’ element of the set K using $n$-dimensional subspaces of X. We will show that some Hölder widths are essentially smaller than $n$-Kolmogorov widths.

Theorem 5.

For a compact set $K\subset X$ and $n\ge 1$, $0<\alpha <1$,

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\le d_n{\left(K\right)}_X,for\gamma ={\displaystyle \frac{d_n{\left(K\right)}_X+\mathrm{rad}\left(K\right)}{2^{\alpha -1}}}.$$

Corollary 1.

If $K\subset X$ is compact, then for each $n_0\in \mathbb{N}$ and each $\gamma \ge {\displaystyle \frac{d_{n_0}{\left(K\right)}_X+\mathrm{rad}\left(K\right)}{2^{\alpha -1}}}$,

$$\underset{n\to \infty }{lim}{\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X=0.$$

Remark 4.

Recall that the linear width $d_n^L{\left(K\right)}_X$ is defined as

$$d_0^L{\left(K\right)}_X:=\underset{f\in K}{sup}{\parallel f\parallel }_X,d_n^L{\left(K\right)}_X:=\underset{L\in {\mathcal{L}}_n}{inf}\underset{f\in K}{sup}{\parallel f-L\left(f\right)\parallel }_X,n\ge 1,$$

where the infimum is taken over the class ${\mathcal{L}}_n$ of all continuous linear operators from X into itself with rank at most n. It follows from the definitions of the $n$-Kolmogorov width and linear width that $d_n{\left(K\right)}_X\le d_n^L{\left(K\right)}_X$. Thus, Theorem 5 implies that

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\le d_n^L{\left(K\right)}_X,for\gamma ={\displaystyle \frac{d_n{\left(K\right)}_X+\mathrm{rad}\left(K\right)}{2^{\alpha -1}}}.$$

3.2. The Relationship Between Hölder Widths and Nonlinear $(n,N)$-Widths

To evaluate the performance of the best $n$-term approximation with respect to different systems, such as the trigonometric system and wavelet bases, Temlyakov introduced the nonlinear $(n,N)$-width in [16], which is defined as follows: for $N\ge 1$, $n\ge 1$,

$$d_0{(K,N)}_X:=\underset{f\in K}{sup}{\parallel f\parallel }_X,d_n{(K,N)}_X:=\underset{{\mathcal{L}}_N}{inf}\underset{f\in K}{sup}\underset{X_n\in {\mathcal{L}}_N}{inf}\mathrm{dist}{(f,X_n)}_X.$$

(6)

where the second infimum is taken over the sets ${\mathcal{L}}_N$ of N linear spaces $X_n\subset X$ of dimension n. The nonlinear $(n,N)$-width can reflect the approximation performance of greedy algorithms.

It is clear that $d_n{(K,1)}_X=d_n{\left(K\right)}_X$. The larger N is, the more flexibility we have in approximating f. Moreover, it is known from (6) and Theorem 5 that

$$d_n{(K,N)}_X\ge d_{n·N}{\left(K\right)}_X\ge {\delta }_{n·N}^{\gamma ,\alpha }{\left(K\right)}_X,\mathrm{where}\gamma =2^{2-\alpha }\mathrm{rad}\left(K\right).$$

Moreover, we obtain the following inequalities, revealing the relationship between Hölder widths and nonlinear $(n,N)$-widths.

Theorem 6.

For any $n\ge 1$, $N>1$, $0<\alpha <1$, and any compact set $K\subset X$ with $\underset{f\in K}{sup}{\parallel f\parallel }_X=1$, we have

$${\delta }_{n+1}^{4(N+1)\sqrt{n},\alpha }{\left(K\right)}_X\le d_n{(K,N)}_X,\mathrm{and}{\delta }_{n+\lceil {log}_{2}N\rceil }^{12\sqrt{n},\alpha }{\left(K\right)}_X\le d_n{(K,N)}_X.$$

4. Comparison Between Hölder Widths and Entropy Numbers

We first recall the definition of the entropy number from [29]. The entropy number ${\epsilon }_n{\left(K\right)}_X$ is defined as

$${\epsilon }_n{\left(K\right)}_X:=inf\{\epsilon >0:K\subset \bigcup _{j=1}^{2^n}B(g_j,\epsilon ),g_j\in X,j=1,2,\cdots ,2^n\},$$

which is the infimum of all $\epsilon >0$ for which $2^n$ balls of radius $\epsilon$ cover a compact set $K\subset X$, and $B(g_j,\epsilon ):=\{g\in X:\parallel g-g_j{\parallel }_X\le \epsilon \}$.

Entropy numbers have many applications in fields such as compressed sensing, statistics, and learning theory [30,31,32]. They can provide a benchmark for the best error performance of numerical recovery algorithms. Sometimes, estimating the entropy numbers ${\epsilon }_n{\left(K\right)}_X$ is more accessible than computing other known widths, such as $n$-Kolmogorov widths and nonlinear $(n,N)$-widths. For example, for some model classes K, such as unit balls in classical Sobolev and Besov spaces, the entropy numbers are known and can also be used to estimate the lower bound of these widths.

In this section, we compare the convergence rate of the Hölder widths with that of the entropy numbers. We first obtain the following general results.

Theorem 7.

For any $n\ge 1$ and $0<\alpha <1$, we have

$${\delta }_n^{2^k\mathrm{rad}\left(K\right),\alpha }{\left(K\right)}_X\le {\epsilon }_{kn}{\left(K\right)}_X,k=1,2,\cdots .$$

Specifically, if $k=n$, then

$${\delta }_n^{2^n\mathrm{rad}\left(K\right),\alpha }{\left(K\right)}_X\le {\epsilon }_{n^2}{\left(K\right)}_X.$$

Remark 5.

It is known from Theorem 7 that if $k=1$, then

$${\delta }_n^{2·\mathrm{rad}\left(K\right),\alpha }{\left(K\right)}_X\le {\epsilon }_n{\left(K\right)}_X.$$

So, by the decreasing property of ${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X$ with respect to γ, Hölder widths are smaller than entropy numbers for $\gamma \ge 2·\mathrm{rad}\left(K\right)$.

It follows from Theorem 7 with $k=1$ that the following corollary holds.

Corollary 2.

Let $0<\alpha <1$, and $\gamma \ge 2·\mathrm{rad}\left(K\right)$.

(i) If the following inequality holds:

$${\epsilon }_n{\left(K\right)}_X\le c_1{\displaystyle \frac{{\left({log}_{2}n\right)}^p}{n^q}},n\ge 1,$$

where $c_1,p,q$ are some constants such that $p\in \mathbb{R}$, $c_1,q>0$, then we have

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\le C{\displaystyle \frac{{\left({log}_{2}n\right)}^p}{n^q}},n\ge 1,$$

where C is a positive constant.

(ii) If the following inequality holds:

$${\epsilon }_n{\left(K\right)}_X\le c_1{\left({log}_{2}n\right)}^{-q},n\ge 1,$$

where $c_1,q$ are two positive constants, then we have

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\le C{\left({log}_{2}n\right)}^{-q},n\ge 1,$$

where C is a positive constant.

Theorem 7 and Corollary 2 state that given an upper bound on the entropy number, it can be determined that the upper bound of the Hölder width is less than it. Conversely, if we know a lower bound of the entropy number, then we can obtain a lower bound of the Hölder width. To show this, we need the following theorem.

Theorem 8.

Let $\gamma >0$, $0<\alpha <1$. If there exists $n>n_0$, where $n_0=n_0(c_0,\alpha ,p,q)$ satisfies $p\in \mathbb{R},q>0$,

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X<c_0{\displaystyle \frac{{\left({log}_{2}n\right)}^p}{n^q}},$$

then for $m=c_{1}n{log}_{2}n$,

$${\epsilon }_m{\left(K\right)}_X<C{\displaystyle \frac{{\left({log}_{2}m\right)}^{p+q}}{m^q}},$$

where $c_1,C$ are constants that depend on $\gamma ,\alpha ,c_0,p$, and q.

Based on Theorem 8, we can obtain a lower bound of the Hölder width from that of the entropy number.

Theorem 9.

(i) If the following inequality holds:

$${\epsilon }_n{\left(K\right)}_X>c_1{\displaystyle \frac{{\left({log}_{2}n\right)}^p}{n^q}},n\ge 1,$$

where $c_1,p,q$ are constants such that $p\in \mathbb{R}$, $c_1,q>0$, then for each $\gamma >0$ and $0<\alpha <1$, we have

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\ge C{\displaystyle \frac{{\left({log}_{2}n\right)}^{p-q}}{n^q}},n\ge 1,$$

(7)

where C is a positive constant.

(ii) If the following inequality holds:

$${\epsilon }_n{\left(K\right)}_X>c_1{\left({log}_{2}n\right)}^{-q},n\ge 1,$$

where $c_1,q$ are some positive constants, then for each $\gamma >0$ and $0<\alpha <1$, we have

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\ge C{\left({log}_{2}n\right)}^{-q},n\ge 1,$$

(8)

where C is a positive constant.

(iii) If the following inequality holds:

$${\epsilon }_n{\left(K\right)}_X>c_1{2}^{-c_2{n}^p},n\ge 1,$$

where $c_1,c_2,p$ are some constants such that $c_1,c_2>0$, and $0<p<1$, then for each

$$\gamma >{\displaystyle \frac{d_n{\left(K\right)}_X+\mathrm{rad}\left(K\right)}{2^{\alpha -1}}},\mathrm{and}0<\alpha <1,$$

we have

$${\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\ge C_1{2}^{-C_2{n}^{{\displaystyle \frac{p}{1-p}}}},n\ge 1,$$

(9)

where $C_1,C_2$ are two positive constants.

Combining Corollary 2 (ii) with Theorem 9 (ii), we derive the following corollary.

Corollary 3.

For any compact set $K\subset X$, $0<\alpha <1$, $\gamma \ge max\left\{2·\mathrm{rad}\left(K\right),{\displaystyle \frac{d_{n_0}{\left(K\right)}_X+\mathrm{rad}\left(K\right)}{2^{\alpha -1}}}\right\}$, and $q>0$, then

$${\epsilon }_n{\left(K\right)}_X\asymp {\left({log}_{2}n\right)}^{-q}⟺{\delta }_n^{\gamma ,\alpha }{\left(K\right)}_X\asymp {\left({log}_{2}n\right)}^{-q},\mathrm{for}\mathrm{any}n\ge 1.$$

5. Some Applications

The importance of the Hölder width lies in its lower bound, which is independent of any specific algorithm. This bound not only reveals the limitations of certain approximation tools but also provides information on the order of the Hölder width without knowing the concrete algorithms. The insights from the lower bound can help us show the optimality of some existing algorithms or prompt us to design optimal algorithms that can achieve such a bound. In essence, the concept of width is independent of any specific algorithm but inspires us to design optimal algorithms.

We apply the above general theoretical results to some important function spaces and obtain the corresponding orders of Hölder widths.

First, we remark that some common neural networks are Hölder mappings. Therefore, we can obtain their asymptotic orders of Hölder widths characterized by these fully connected feed-forward neural networks. In the following discussion, we mainly consider the Banach spaces X of functions, where $C\left({[0,1]}^d\right)$ is continuously embedded in X. Let $\sigma :\mathbb{R}\to \mathbb{R}$ be an activation function. Denote by $\mathsf{\Omega }:={[0,1]}^d$ and ${\mathsf{\Phi }}_{\sigma }:\left(B_{l_{\infty }^{\tilde{n}}},{\parallel ·\parallel }_{l_{\infty }^{\tilde{n}}}\right)\to C\left(\mathsf{\Omega }\right)$.

A feed-forward neural network with width W, depth n, and activation $\sigma$ produces a family ${\mathsf{\Sigma }}_{n,\sigma }\subset C\left(\mathsf{\Omega }\right)$:

$${\mathsf{\Sigma }}_{n,\sigma }:=\left\{{\mathsf{\Phi }}_{\sigma }\left(t\right):t\in {\mathbb{R}}^{\tilde{n}},\tilde{n}=\tilde{n}(W,n)=C_{0}n\right\},$$

which generates an approximation to a target element $f\in K$. For every $t\in {\mathbb{R}}^{\tilde{n}},$ there exists a continuous function ${\mathsf{\Phi }}_{\sigma }\left(t\right)\in {\mathsf{\Sigma }}_{n,\sigma }$ on $\mathsf{\Omega }$

$${\mathsf{\Phi }}_{\sigma }\left(t\right):=T^{\left(n\right)}\circ \overline{\sigma }\circ T^{(n-1)}\circ \overline{\sigma }\circ \cdots \circ \overline{\sigma }\circ T^{\left(0\right)},$$

(10)

where the affine mappings $T^{\left(0\right)}:{\mathbb{R}}^d\to {\mathbb{R}}^W$, $T^{\left(k\right)}:{\mathbb{R}}^W\to {\mathbb{R}}^W$, $k=1,2\cdots ,n-1$, and $T^{\left(n\right)}:{\mathbb{R}}^W\to \mathbb{R}$, and the function $\overline{\sigma }:{\mathbb{R}}^W\to {\mathbb{R}}^W$

$$\overline{\sigma }(z_{j+1},\cdots ,z_{j+W}):=\left(\sigma \left(z_{j+1}\right),\cdots ,\sigma \left(z_{j+W}\right)\right).$$

Here, t is the vector whose coordinates are the entries of the matrices and biases of $T^{\left(k\right)}$, $k=0,1,\cdots ,n$. We note that the dimension of any hidden layer can naturally be expanded; thus, any fully connected network can be made to have a fixed width [13,33]. Our assumption about a fixed width W can simplify the computations and notations.

Proposition 1.

If σ is an $H_{\alpha }\left(\gamma \right)$ mapping, then ${\mathsf{\Phi }}_{\sigma }:\left(B_{l_{\infty }^{\tilde{n}}},{\parallel ·\parallel }_{l_{\infty }^{\tilde{n}}}\right)$, defined in (10), is an $H_{\alpha }\left({\mathsf{\Gamma }}_n\right)$ mapping, which means that for $t,t^{\prime }\in B_{l_{\infty }^{\tilde{n}}}$,

$$\parallel {\mathsf{\Phi }}_{\sigma }\left(t\right)-{\mathsf{\Phi }}_{\sigma }\left(t^{\prime }\right){\parallel }_{C\left(\mathsf{\Omega }\right)}\le {\mathsf{\Gamma }}_n{\parallel t-t^{\prime }\parallel }_{l_{\infty }^{\tilde{n}}}^{\alpha },$$

where ${\mathsf{\Gamma }}_n=M_{d,\alpha }{\gamma }_0^n{\left(2W\right)}^{n\alpha }$, ${\gamma }_0=max\{\gamma ,1\}$, and $M_{d,\alpha }$ is a constant.

It follows that a specific neural network can be a Hölder mapping with coefficient ${\mathsf{\Gamma }}_n=M_{d,\alpha }{\left({\left(2W\right)}^{\alpha }{\gamma }_0\right)}^n$ to approximate the target element f, where $0<\alpha <1$ and $W>1$. Then, we consider the lower bound for the Hölder width with coefficient ${\gamma }_n=M{\lambda }^n$, $M>0$, $\lambda >1$, which also implies the lower bound of the DNN approximation.

Theorem 10.

Let $0<\alpha <1$ and ${\gamma }_n=M{\lambda }^n$, where $M>0$, $\lambda >1$, $n\ge 1$.

(i) If the following inequality holds:

$${\epsilon }_n{\left(K\right)}_X>c_1{\displaystyle \frac{{\left({log}_{2}n\right)}^p}{n^q}},n\ge 1,$$

where $c_1,p,q$ are constants such that $p\in \mathbb{R}$, $c_1,q>0$, then

$${\delta }_n^{{\gamma }_n,\alpha }{\left(K\right)}_X\ge C{\displaystyle \frac{{\left({log}_{2}n\right)}^p}{n^{2q}}},n\ge 1,$$

(11)

where C is a positive constant.

(ii) If the following inequality holds:

$${\epsilon }_n{\left(K\right)}_X>c_1{\left({log}_{2}n\right)}^{-q},n\ge 1,$$

where $c_1,q$ are some positive constants, then

$${\delta }_n^{{\gamma }_n,\alpha }{\left(K\right)}_X\ge C{\left({log}_{2}n\right)}^{-q},n\ge 1,$$

(12)

where C is a positive constant.

Theorems 7 and 10 imply the following corollary, which means that if we know the asymptotic orders of some entropy numbers, we can obtain the asymptotic orders of their Hölder widths.

Corollary 4.

Let $0<\alpha <1$ and ${\gamma }_n=M{\lambda }^n$, where $M>0$, $\lambda >1$, $n\ge 1$.

(i) If the following holds:

$${\epsilon }_n{\left(K\right)}_X\asymp {\displaystyle \frac{{\left({log}_{2}n\right)}^p}{n^q}},n\ge 1,$$

where $p,q$ are some constants such that $p\in \mathbb{R}$, $q>0$, then we have

$${\delta }_n^{{\gamma }_n,\alpha }{\left(K\right)}_X\asymp {\displaystyle \frac{{\left({log}_{2}n\right)}^p}{n^{2q}}},n\ge 1.$$

(ii) If the following holds:

$${\epsilon }_n{\left(K\right)}_X\asymp {\left({log}_{2}n\right)}^{-q},n\ge 1,$$

where q is a positive constant, then we have

$${\delta }_n^{{\gamma }_n,\alpha }{\left(K\right)}_X\asymp {\left({log}_{2}n\right)}^{-q},n\ge 1.$$

We point out that Corollary 4 provides a tool for giving lower bounds on how well a compact set K can be approximated by a DNN, where all weights and biases are from the unit ball of some norm ${\parallel ·\parallel }_{Y_{\tilde{n}}}$. The classical model classes K for multivariate functions can be the unit balls of smoothness spaces, such as classical Lipschitz, Hölder, Sobolev, and Besov spaces. For any model class K, denote the unit ball of K by

$$BK:=\{f\in K:\parallel f{\parallel }_K\le 1\}.$$

For any $K,A\subset X$, let

$$\mathrm{dist}{\left(K,A\right)}_X:=\underset{f\in K}{sup}\underset{g\in A}{inf}{\parallel f-g\parallel }_X.$$

Many experts have investigated the performance of deep learning approximation for these function classes of the Lebesgue space $L_p$ on the cube $\mathsf{\Omega }$ [20,21,33,34].

First, we determine the exact order of the Hölder width for the classical Sobolev class. For $1\le p\le \infty$, we denote by $L_p\left(\mathsf{\Omega }\right)$ the usual Lebesgue space on $\mathsf{\Omega }$ equipped with the $L_p$ norm

$${\parallel f\parallel }_p:={\parallel f\parallel }_{L_p\left(\mathsf{\Omega }\right)}=\left\{\begin{array}{cc}& {\left({\int }_{\mathsf{\Omega }}{\left|f\left(x\right)\right|}^p\mathrm{d}\mu \left(x\right)\right)}^{1/p},1\le p<\infty ,\hfill \\ & \mathrm{ess}\underset{x\in \mathsf{\Omega }}{sup}\left|f\left(x\right)\right|,p=\infty .\hfill \end{array}\right.$$

For $s\in \mathbb{N}$, $1\le p\le \infty$, we say that function f belongs to the Sobolev space $W_p^s\left(\mathsf{\Omega }\right)$ if $f\in L_p\left(\mathsf{\Omega }\right)$, and the norm ${\parallel ·\parallel }_{W_p^s}$ of f is given by, for $\left|\mathbf{k}\right|:={\sum }_{j=1}^d\left|k_j\right|=s$,

$${\parallel f\parallel }_{W_p^s}:={\parallel f\parallel }_p+\underset{\left|\mathbf{k}\right|=s}{max}\parallel D^{\mathbf{k}}f{\parallel }_p<\infty .$$

It is known from [35] that when $s>d{(1/p-1/q)}_{+}$, $1<p,q<\infty$

$${\epsilon }_n{\left(B{W}_p^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\asymp n^{-s/d}.$$

(13)

Moreover, the approximation error rates $\mathcal{O}\left(n^{-s/d}\right)$ for Sobolev and Besov classes can be obtained using many classical methods of nonlinear approximation, such as adaptive finite elements or $n$-term wavelet approximation [13,15].

It follows from Theorem 10 and Corollary 4 that for some deep neural networks,

$$\mathrm{dist}{\left(B{W}_p^s\left(\mathsf{\Omega }\right),{\mathsf{\Sigma }}_{n,\sigma }\right)}_{L_q}\ge {\delta }_{\tilde{n}}^{{\gamma }_n,\alpha }{\left(B{W}_p^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\gg n^{-2s/d},$$

(14)

where ${\mathsf{\Sigma }}_{n,\sigma }$ is produced by some neural networks with depth n, fixed width, and $H_{\alpha }\left(\gamma \right)$ activation functions $\sigma$, $0<\alpha \le 1$. Compared with classical methods, the factor 2 in the exponent leaves open the possibility of improved approximation rates when using deep neural networks. Indeed, it is known from [33] that there exists a neural network ${\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}$ with depth n, width $25d+31$, and ReLU activation function ${\sigma }^{\prime }$ such that, for $f\in B{W}_p^s\left(\mathsf{\Omega }\right),$

$$\underset{f_n\in {\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}}{inf}\parallel f-f_n{\parallel }_q\ll {\parallel f\parallel }_{W_p^s}·n^{-2s/d},$$

(15)

The author used a novel bit-extraction technique, which gives an optimal encoding of sparse vectors, to obtain the upper bounds $\mathcal{O}\left(n^{-2s/d}\right)$.

Based on the above discussion, we obtain the Hölder widths for the Sobolev classes $B{W}_p^s\left(\mathsf{\Omega }\right)$.

Theorem 11.

Let $1<p,q<\infty$, $0<\alpha \le 1$, and $s>d{\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{q}}\right)}_{+}$. Then, there exist $M>0$, $\lambda >1$ such that for ${\gamma }_n=M{\lambda }^n$,

$${\delta }_n^{{\gamma }_n,\alpha }{\left(B{W}_p^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\asymp \mathrm{dist}{\left(B{W}_p^s\left(\mathsf{\Omega }\right),{\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}\right)}_{L_q}\asymp n^{-2s/d}.$$

Proof. 

It is known from (13) and Corollary 4 that for any $M>0$, $\lambda >1$,

$${\delta }_n^{{\gamma }_n,\alpha }{\left(B{W}_p^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\gg n^{-2s/d}.$$

Moreover, it is known from (14) that

$$\mathrm{dist}{\left(B{W}_p^s\left(\mathsf{\Omega }\right),{\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}\right)}_{L_q}\gg n^{-2s/d}.$$

It follows from (15) that

$$\mathrm{dist}{\left(B{W}_p^s\left(\mathsf{\Omega }\right),{\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}\right)}_{L_q}\asymp \underset{f\in B{W}_p^s\left(\mathsf{\Omega }\right)}{sup}\underset{f_n\in {\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}}{inf}{\parallel f-f_n\parallel }_q\ll n^{-2s/d},$$

and for any $f\in B{W}_p^s\left(\mathsf{\Omega }\right),$

$$\underset{f_n\in {\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}}{inf}{\parallel f-f_n\parallel }_q\ll n^{-2s/d}.$$

Therefore, there exist constants c, $c_0>0$ such that

$${\delta }_{c_{0}n}^{c{(50d+62)}^n,1}{\left(B{W}_p^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\ll n^{-2s/d}.$$

Replacing $c_{0}n$ with n, it follows from the decreasing property of the Hölder width that there exist $M=2c>0$ and $\lambda ={(50d+62)}^{1/c_0}>1$ such that

$${\delta }_n^{{\gamma }_n,\alpha }{\left(B{W}_p^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\le {\delta }_n^{c{\lambda }^n,1}{\left(B{W}_p^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\ll n^{-2s/d}.$$

Thus, we complete the proof of Theorem 11. □

Remark 6.

Theorem 11 implies that the upper bound in inequality (15) is sharp.

For a fixed $s\in \mathbb{N}$ and $d=1$, Figure 1 compares the performance of the approximation error versus the number of elements or layers using different approximation methods, such as the classical methods represented by $n$-term wavelets, adaptive finite elements, and new tools represented by deep neural networks. The blue solid line shows the approximation error decreasing at a rate of $\mathcal{O}\left(n^{-s}\right)$ with the number of elements n for $n$-term wavelets or adaptive finite elements, while the orange dashed line indicates a faster decay of $\mathcal{O}\left(n^{-2s}\right)$ with depth n for deep neural networks. Overall, deep neural networks significantly outperform classical methods, such as $n$-term wavelets or adaptive finite elements, offering more rapid convergence and potentially higher accuracy in approximating functions. We call this phenomenon the super-convergence of deep neural networks, where the classical Hölder, Sobolev, and Besov classes on ${[0,1]}^d$ can achieve super-convergence [20,21].

The results of Theorem 11 can be extended to Besov spaces, which are much more general than Sobolev spaces. It is well known that functions from Besov spaces have been widely used in approximation theory, statistics, image processing, and machine learning (see [6,36,37,38], and the references therein). Recall that for $r\in \mathbb{N}$, the modulus of smoothness of order r of $f\in L_p\left(\mathsf{\Omega }\right)$ is

$${\omega }_r{(f,t)}_p:=\underset{\left|h\right|\le t}{sup}{\parallel {▵}_h^r(f,·,\mathsf{\Omega })\parallel }_p,$$

where ${▵}_h^r(f,x)$ is the r-th order difference of f with step h, and

$${▵}_h^r(f,x,\mathsf{\Omega }):=\left\{\begin{array}{cc}& {▵}_h^r(f,x),x,x+h,\cdots ,x+rh\in \mathsf{\Omega },\hfill \\ & 0,\mathrm{otherwise}.\hfill \end{array}\right.$$

For $0<s<r$, $1\le p,\tau \le \infty$, we say that function f belongs to the Besov space $B_{p,\tau }^s\left(\mathsf{\Omega }\right)$ if $f\in L_p\left(\mathsf{\Omega }\right)$, and the norm ${\parallel ·\parallel }_{B_{p,\tau }^s}$ of f, given by

$$\begin{array}{c}{\parallel f\parallel }_{B_{p,\tau }^s}:=\left\{\begin{array}{cc}& {\parallel f\parallel }_p+{\left({\int }_0^1{\left({\displaystyle \frac{{\omega }_r{(f,t)}_p}{t^s}}\right)}^{\tau }{\displaystyle \frac{\mathrm{d}t}{t}}\right)}^{{\displaystyle \frac{1}{\tau }}},1\le \tau <\infty ;\hfill \\ & {\parallel f\parallel }_p+\underset{t>0}{sup}{\displaystyle \frac{{\omega }_r{(f,t)}_p}{t^s}},\tau =\infty ,\hfill \end{array}\right.\end{array}$$

is finite. It is known from [15] that when $s>d{(1/p-1/q)}_{+}$, $q\ne \infty$

$${\epsilon }_n{\left(B{B}_{p,\tau }^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\asymp n^{-s/d}.$$

It follows from Theorem 10 and Corollary 4 that for some deep neural networks,

$$\mathrm{dist}{\left(B{B}_{p,\tau }^s\left(\mathsf{\Omega }\right),{\mathsf{\Sigma }}_{n,\sigma }\right)}_{L_q}\ge {\delta }_{\tilde{n}}^{{\gamma }_n,\alpha }{\left(B{B}_{p,\tau }^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\gg n^{-2s/d},$$

and it is also known from [33] that there exists a neural network ${\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}$ with depth n, width $25d+31$, and ReLU activation function ${\sigma }^{\prime }$ such that for $f\in B{B}_{p,\tau }^s\left(\mathsf{\Omega }\right),$

$$\underset{f_n\in {\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}}{inf}\parallel f-f_n{\parallel }_q\ll {\parallel f\parallel }_{B_{p,\tau }^s}·n^{-2s/d}.$$

Thus, we obtain the Hölder widths for the Besov classes $B{B}_{p,\tau }^s\left(\mathsf{\Omega }\right)$.

Theorem 12.

Let $1<p,q<\infty$, $1\le \tau \le \infty$, $0<\alpha \le 1$, and $s>d{\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{q}}\right)}_{+}$. Then, there exist $M>0$, $\lambda >1$ such that for ${\gamma }_n=M{\lambda }^n$,

$${\delta }_n^{{\gamma }_n,\alpha }{\left(B{B}_{p,\tau }^s\left(\mathsf{\Omega }\right)\right)}_{L_q}\asymp \mathrm{dist}{\left(B{B}_{p,\tau }^s\left(\mathsf{\Omega }\right),{\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}\right)}_{L_q}\asymp n^{-2s/d}.$$

The proof of Theorem 12 is similar to that of Theorem 11, so we omit the details here.

Note that any numerical algorithm based on Hölder mappings will have a convergence rate that is not faster than that of the Hölder width. This characterizes the limitation of the approximation power of some deep neural networks.

Meanwhile, it is well known that spherical approximation has been widely applied in many fields, such as cosmic microwave background analysis, global ionospheric prediction for geomagnetic storms, climate change modeling, environmental governance, and other spherical signals [39].

We recall some concepts on the sphere from [40]. Let

$${\mathbb{S}}^{d-1}:=\left\{x=(x_1,\cdots ,x_d)\in {\mathbb{R}}^d:\sum _{j=1}^d{x}_j^2=1\right\}$$

be the unit sphere in ${\mathbb{R}}^d$ equipped with the rotation-invariant measure $\mathrm{d}\mu \left(x\right)$ normalized by ${\int }_{{\mathbb{S}}^{d-1}}\mathrm{d}\mu \left(x\right)=1$. For $1\le p<\infty$, denote by $L_p\left({\mathbb{S}}^{d-1}\right)$ the usual Lebesgue space on ${\mathbb{S}}^{d-1}$ endowed with the $L_p$ norm

$${\parallel f\parallel }_{L_p\left({\mathbb{S}}^{d-1}\right)}:={\left({\int }_{{\mathbb{S}}^{d-1}}{\left|f\left(x\right)\right|}^p\mathrm{d}\mu \left(x\right)\right)}^{1/p}\mathrm{for}1\le p<\infty .$$

In [41], Feng et al. studied the approximation of the Sobolev class on the sphere ${\mathbb{S}}^{d-1}$, denoted by $W_p^s\left({\mathbb{S}}^{d-1}\right)$, using convolutional neural networks with layer J. Recall that a function f belongs to the Sobolev class $W_p^s\left({\mathbb{S}}^{d-1}\right)$ if $f\in L_p\left({\mathbb{S}}^{d-1}\right)$, and the norm ${\parallel ·\parallel }_{W_p^s\left({\mathbb{S}}^{d-1}\right)}$ of f is given by

$${\parallel f\parallel }_{W_p^s\left({\mathbb{S}}^{d-1}\right)}:={\parallel f\parallel }_{L_p\left({\mathbb{S}}^{d-1}\right)}+\parallel {(-{▵}_0)}^{s/2}f{\parallel }_{L_p\left({\mathbb{S}}^{d-1}\right)}<\infty ,$$

where ${▵}_0$ is the Laplace–Beltrami operator on the sphere. The authors obtained the upper bound $\mathcal{O}\left(J^{-{\displaystyle \frac{s}{d-1}}}\right)$ of the error of such an approximation in $L_p\left({\mathbb{S}}^{d-1}\right)$.

However, it is known from [42] that when $s>(d-1){(1/p-1/q)}_{+}$, $1<p,q<\infty$

$${\epsilon }_n{\left(B{W}_p^s\left({\mathbb{S}}^{d-1}\right)\right)}_{L_q}\asymp n^{-{\displaystyle \frac{s}{d-1}}}.$$

Then, it follows from Corollary 4 that the lower bound of the Hölder width for $B{W}_p^s\left({\mathbb{S}}^{d-1}\right)$ may be $\mathcal{O}\left(n^{-{\displaystyle \frac{2s}{d-1}}}\right)$.

Theorem 13.

Let $1<p,q<\infty$, $0<\alpha \le 1$, and $s>(d-1){\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{q}}\right)}_{+}$. Then, there exist $M>0$, $\lambda >1$ such that for ${\gamma }_n=M{\lambda }^n$,

$$\mathrm{dist}{\left(B{W}_p^s\left({\mathbb{S}}^{d-1}\right),{\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}\right)}_{L_q}\gg {\delta }_n^{{\gamma }_n,\alpha }{\left(B{W}_p^s\left({\mathbb{S}}^{d-1}\right)\right)}_{L_q}\gg n^{-{\displaystyle \frac{2s}{d-1}}}.$$

With the development of spherical theory [40,43,44,45] and the relationship between Hölder widths and entropy numbers, we conjecture that the approximation order of $W_p^s\left({\mathbb{S}}^{d-1}\right)$ using some fully connected feed-forward neural networks with the bit-extraction technique may be $\mathcal{O}(n^{-{\displaystyle \frac{2s}{d-1}}})$. This conjecture can be formulated as follows.

Conjecture 1.

Let $1<p,q<\infty$, $0<\alpha \le 1$, and $s>(d-1){\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{q}}\right)}_{+}$. Then, there exist $M>0$, $\lambda >1$ such that for ${\gamma }_n=M{\lambda }^n$,

$${\delta }_n^{{\gamma }_n,\alpha }{\left(B{W}_p^s\left({\mathbb{S}}^{d-1}\right)\right)}_{L_q}\asymp \mathrm{dist}{\left(B{W}_p^s\left({\mathbb{S}}^{d-1}\right),{\mathsf{\Sigma }}_{n,{\sigma }^{\prime }}\right)}_{L_q}\asymp n^{-{\displaystyle \frac{2s}{d-1}}}.$$

Our results show that networks modeled by both Hölder and Lipschitz (e.g., ReLU) functions can achieve the approximation error $\mathcal{O}\left(n^{-2s/d}\right)$, which is superior to classical approximation tools such as $n$-term wavelets and adaptive finite elements. We achieve the same approximation error for networks with a weaker condition, $0<\alpha <1$, which gives us more options for neural network approximation tools. For example, if we need to solve numerically differential equations with a discontinuous right-hand side modeling neural network dynamics, we can choose networks with non-Lipschitz activation functions, using the fact that non-Lipschitz functions share the peculiar property that even small variations in the neuron state are able to produce significant changes in the neuron output [26,27,28]. The results of the Hölder widths would help us select suitable Hölder activation functions. Moreover, it is known from Appendix B.2 that the Hölder width is smaller than the Lipschitz width in the sense that

$${\delta }_n^{2\gamma ,\alpha }{\left(K\right)}_X\le {\delta }_n^{\gamma ,1}{\left(K\right)}_X.$$

Thus, from the perspective of the approximation error, the Hölder activation function performs better than the Lipschitz activation function. However, it is currently unknown what magnitude this improvement can reach. It would be interesting to study this problem.

Next, we estimate the Hölder widths for discrete Lebesgue spaces. For $1\le q<\infty$, denote by ${\ell }_q$ the set of all sequences ${\left\{x_k\right\}}_{k=1}^{\infty }$ with

$$\begin{array}{c}\hfill {\parallel x\parallel }_q:={\left(\sum _{k=1}^{\infty }{\left|x_k\right|}^q\right)}^{1/q}<\infty .\end{array}$$

Let $\tau ={\left\{t_k\right\}}_{k=1}^{\infty }$ be a sequence such that $t_k={\left({log}_2(k+1)\right)}^{-1/2}$ for $k\ge 1$. Denote by

$${\mathcal{A}}_{\tau }:=\left\{y\in {\ell }_2:y_k=t_k{x}_k,\mathrm{where}\sum _{j=1}^{\infty }\left|x_k\right|\le 1\right\},$$

where $x=(x_1,x_2,\cdots )\in {\ell }_1$, and $y=(y_1,y_2,\cdots )\in {\ell }_2$.

Theorem 14.

For the space $X={\ell }_2$ and the subset ${\mathcal{A}}_{\tau }$, its Hölder width satisfies $\gamma \ge 2·\mathrm{rad}\left({\mathcal{A}}_{\tau }\right)$ and $0<\alpha <1$,

$${\left(logn\right)}^{-1/2}{n}^{-1/2}\ll {\delta }_n^{\gamma ,\alpha }{\left({\mathcal{A}}_{\tau }\right)}_X\ll n^{-1/2}.$$

Proof. 

It is known from [46] that its entropy number satisfies

$${\epsilon }_n{\left({\mathcal{A}}_{\tau }\right)}_{{\ell }_2}\asymp n^{-1/2}.$$

It follows from Corollary 2 that there exists $C_1>0$ such that its Hölder width satisfies $\gamma \ge 2·\mathrm{rad}\left({\mathcal{A}}_{\tau }\right)$,

$${\delta }_n^{\gamma ,\alpha }{\left({\mathcal{A}}_{\tau }\right)}_{{\ell }_2}\le C_1{n}^{-1/2}.$$

It follows from Theorem 9 (i) that there exists $C_2>0$ such that its Hölder width satisfies

$${\delta }_n^{\gamma ,\alpha }{\left({\mathcal{A}}_{\tau }\right)}_{{\ell }_2}\ge C_2{\left(logn\right)}^{-1/2}{n}^{-1/2}.$$

The proof of Theorem 14 is completed. □

Remark 7.

It is known from [18] that its $n$-Kolmogorov width satisfies

$$d_n{\left({\mathcal{A}}_{\tau }\right)}_{{\ell }_2}\asymp {\left({log}_{2}n\right)}^{-1/2}.$$

Theorem 14 illustrates that the Hölder width of ${\mathcal{A}}_{\tau }$ is smaller than that of the $n$-Kolmogorov width.

Finally, we obtain the asymptotic order of the Hölder width for ${\mathbf{c}}_{\mathbf{0}}$, which is the Banach space of all sequences converging to 0, equipped with the ${\ell }_{\infty }$ norm. Let $\eta ={\left\{{\eta }_k\right\}}_{k=1}^{\infty }$ be a sequence with ${\eta }_k={\left({log}_2(k+1)\right)}^{-1}$, $k\ge 1$. Denote a compact subset of X by

$${\mathcal{K}}_{\eta }:={\left\{{\eta }_k{e}_k\right\}}_{k=1}^{\infty }\cup \left\{0\right\},$$

where the sequence ${\left\{e_k\right\}}_{k=1}^{\infty }$ is the standard basis in X. It is known from [18] that its entropy number satisfies

$${\epsilon }_n{\left({\mathcal{K}}_{\eta }\right)}_X\asymp {\displaystyle \frac{1}{n}}.$$

Theorem 15.

For the space $X={\mathbf{c}}_{\mathbf{0}}$ and the subset ${\mathcal{K}}_{\eta }$, we obtain that its Hölder width satisfies $\gamma >2$, $0<\alpha <1$,

$${\delta }_n^{\gamma ,\alpha }{\left({\mathcal{K}}_{\eta }\right)}_X\asymp {\displaystyle \frac{1}{n{log}_2(n+1)}}.$$

Theorem 15 shows the sharpness of Theorem 9 (i).

6. Concluding Remarks

We introduce the Hölder width, which measures the best error performance of some recent nonlinear approximation methods. We investigate the relationship between Hölder widths and other known widths, demonstrating that some Hölder widths are essentially smaller than $n$-Kolmogorov widths and linear widths. Moreover, we obtain that as the Hölder constants grow with n, the Hölder widths are much smaller than the entropy numbers. The significance of Hölder widths being smaller than known widths is that it implies that some nonlinear approximations, such as deep neural network approximations, may yield a better approximation order than other known classical approximation methods. In fact, we show that the asymptotic orders of Hölder widths for the Sobolev classes $B{W}_p^s\left(\mathsf{\Omega }\right)$ and the Besov classes $B{B}_{p,\tau }^s\left(\mathsf{\Omega }\right)$ are $\mathcal{O}\left(n^{-2s/d}\right)$ for $s>d{\left({\displaystyle \frac{1}{p}}-{\displaystyle \frac{1}{q}}\right)}_{+}$ using deep neural networks, while other known widths might be $\mathcal{O}\left(n^{-s/d}\right)$. This result shows that deep neural networks significantly outperform classical methods of approximation, such as adaptive finite elements and $n$-term wavelet approximation. Indeed, the Hölder width in neural networks serves two purposes. On the one hand, it demonstrates the superior approximation power of deep neural networks. On the other hand, it reveals the limitation in the approximating ability of some deep neural networks. These features are crucial for a deeper understanding and further exploration of the approximation power of deep neural networks. It would be interesting to calculate the Hölder widths for some important function classes.