Necessary and Sufficient Reservoir Condition for Universal Reservoir Computing

Abstract

We discuss necessary and sufficient conditions for universal approximation using reservoir computing. Reservoir computing is a machine learning method used to train a dynamical system model by tuning only the static part of the model. The universality is the ability of the model to approximate any dynamical system with any precision. In the previous studies, we provided two sufficient conditions for the universality. We employed the universality definition that has been discussed since the earliest studies on reservoir computing. In this present paper, we prove that these two conditions and the universality are equivalent to one another. Using this equivalence, we show that a universal model must have a “pathological” property that can only be achieved or approached by chaotic reservoirs.

1. Introduction

Reservoir computing (RC) is a computationally cheap method used to train dynamical system models, which was initially proposed for recurrent neural networks (RNNs) [1]. Generally, RNNs are trained using the gradient descent method, but this is computationally expensive because the signal flow must be tracked in the network for a certain period. Jaeger et al. [2] and Maass et al. [3] found that RNNs achieve approximation tasks by training only a static function, converting network states to the output. Their methods were unified as RC, a framework that trains dynamical models by training static functions [4,5].

An RC model is a dynamical system model designed to be trained through RC, and it consists of a dynamical system called the “reservoir” and a static function called the “readout.” As shown in Figure 1, the reservoir processes the input to the model first, and the readout maps the reservoir state to the output. In supervised learning, the model aims to approximate the given target system. During RC model training, only the readout is tuned for each target, but the reservoir does not adapt to targets. Empirically, an RC model performs better using a reservoir with complex dynamics. For example, echo state networks [2] use an RNN with random parameters as a reservoir.

Another advantage of RC is that various dynamical systems can be used as the reservoir, even if they are difficult to train or adjust. Recently, physical RC, which uses a reservoir implemented as hardware, has been drawing attention in terms of energy efficiency and computation speed [6,7,8]. Many studies have been conducted on various implementations of reservoirs, e.g., electric and electronic circuits [9,10,11,12,13], network [14] and delay feedback systems [15,16] using optical elements, and spin torque oscillators [17,18,19].

An RC model and its reservoir are said to be universal if the model can approximate an arbitrary target with arbitrary precision. The concept of universality concerning RC appeared simultaneously with RC itself [3]. Maass et al. [3] also proposed a sufficient condition for a continuous-time RC model to be universal. The sufficient condition in [3] is the combination of the continuity and injectivity of the reservoir, which is a functional from input functions to output values. The injectivity of the reservoir is also a necessary condition for universality.

Sugiura et al. [20] proposed the relaxed condition called the neighborhood separation property (NSP), which can be applied to more complex reservoirs with multiple equilibrium states. The authors of [21] showed that a reservoir with a finite-dimensional output can satisfy the NSP but not the condition in [3]. In [21], another sufficient condition was proposed: the existence of the continuous inverse of a reservoir. This condition is also sufficient for the NSP and used to show that the NSP can be satisfied. Relationships among universality and the explained reservoir conditions are summarized in Figure 2.

As mentioned above, some necessary conditions and sufficient conditions for universality are known, but the necessary and sufficient condition, equivalent to universality, is still unknown. Such a condition is critical as it provides an essential answer to the question regarding which properties of the reservoir enable approximation with an RC model. In this paper, we show that the NSP and continuous inverse, which are sufficient conditions for a reservoir to be universal, are also equivalent to the universality itself. Moreover, we also show that a universal reservoir has a “pathological” property using the obtained equivalence. Similarly to previous studies, we considered a continuous-time RC model with a polynomial readout and evaluated the approximation using the maximum error. A dynamical system, e.g., a model, reservoir, and target, is treated as a functional from input functions to output values.

Our result concerning the conditions equivalent to universality can be extended to a general case where the input function space is compactifiable. Hence, our result may be applied to various types of RC not discussed in this paper, such as discrete-time ones.

The pathological property of a universal reservoir is that it has dense discontinuous points. As we show later, a universal reservoir has a continuous inverse map. Hence, if there is a continuous and universal reservoir, it is a homeomorphism. However, the infinite-dimensional space of input functions and the finite-dimensional space of output values cannot be homeomorphic. The same holds if we restrict the reservoir domain to an arbitrary open subset, i.e., a universal reservoir has a discontinuous point in any open subset. This result suggests that a universal reservoir is highly sensitive to inputs, and chaotic reservoirs, such as those described in [19,22,23,24], are necessary to achieve universality. These facts support the empirical rule that a complex reservoir tends to be effective and provide significant insight into the development of high-performance reservoirs.

Although considering noise in inputs and observations is important in practice, we focus on the deterministic and noiseless case for the following reasons. First, theoretical research on continuous-time RC remains limited even in the deterministic setting. Second, the definition of universality in the stochastic case is not straightforward and has not yet been established.

The reminder of this paper is structured as follows: Section 2 provides preliminaries and describes RC and previous results. In Section 3, we prove that the NSP and the continuous inverse of a reservoir are equivalent to universality. In Section 4, we prove that a universal reservoir has dense discontinuous points. The main symbols used in this paper are summarized in

Table 1. Main symbols and their definitions.

Symbol	Definition
${\mathbb{R}}_{-}\subset \mathbb{R}$	$\left(-\infty ,0\right]$
${\mathbb{R}}_{+}\subset \mathbb{R}$	$\left[0,\infty \right)$
${\mathbb{P}}_m$	Set of all polynomial functions from ${\mathbb{R}}^m$ to $\mathbb{R}$
$A\subset {\mathbb{R}}^m$	Compact and convex set of input values
$K>0$	Bound of Lipschitz constants of input functions
V	Set of input functions on ${\mathbb{R}}_{-}$ defined as (1)
${v|}_{·}$	Restriction of map v to set $\left(·\right)$
$V^{\mathrm{res}}$	Set of input functions on a finite time interval defined as (2)
$\overline{·}$	Closure of set $\left(·\right)$
$\overline{V^{\mathrm{res}}}$	Compactification of $V^{\mathrm{res}}$ defined as $\overline{V^{\mathrm{res}}}=V^{\mathrm{res}}\cup V$
$d:\overline{V^{\mathrm{res}}}\times \overline{V^{\mathrm{res}}}\to \mathbb{R}$	Metric on $\overline{V^{\mathrm{res}}}$ defined as (8)
$N_{\delta }\left(v\right)\subset V^{\mathrm{res}}$	$\left\{v^{\prime }\in V^{\mathrm{res}}\left|d\left(v,v^{\prime }\right)<\delta \right.\right\}$ for any $\delta >0$ and $v\in \overline{V^{\mathrm{res}}}$
${\mathrm{id}}_{·}$	Identity mapping on set $\left(·\right)$
$\mathrm{ind}·$	Inductive dimension of set $\left(·\right)$
Symbol (only in Theorem 2)	Definition
X	General compactifiable metric space
$\overline{X}$	Compactification of X
$d:\overline{X}\times \overline{X}\to \mathbb{R}$	Metric on $\overline{X}$
$N_{\delta }\left(x\right)\subset X$	$\left\{x^{\prime }\in X\left|d\left(x,x^{\prime }\right)<\delta \right.\right\}$ for any $\delta >0$ and $x\in \overline{X}$

.

2. Preliminary

We discuss a dynamical system represented as a functional on functions of time. Let $A\subset {\mathbb{R}}^n$ be a compact and convex set of input values and $K>0$ be the limit of the speed of input change. We define the set V of input functions as follows:

$$V=\left\{v:{\mathbb{R}}_{-}\to A\left|\forall t_1,t_2\le 0,∥v\left(t_1\right)-v\left(t_2\right)∥\le K\left|t_1-t_2\right|\right.\right\},$$

(1)

where $∥·∥$ is the Euclidean norm, and ${\mathbb{R}}_{-}$ is defined as $\left(-\infty ,0\right]$. One must set A and $K>0$ large and wide enough according to the input functions that one considers. In reality, such A and K may be unknown or may not exist, but we do not consider these cases here.

Because input functions are given on a finite time interval in practice, we define another set of input functions by restricting the domain of functions in V. For a function v and $t\ge 0$, we write the restriction of v to $\left[-t,0\right]$ as ${v|}_{\left[-t,0\right]}$. We define the input functions on a finite time interval as

$$V^{\mathrm{res}}=\left\{{v|}_{\left[-t,0\right]}\left|v\in V,t\ge 0\right.\right\}.$$

(2)

Note that $V^{\mathrm{res}}$ is not defined for a specific $t\ge 0$ and contains input functions on time intervals of various lengths. The dynamical system that we discuss is a functional from $V^{\mathrm{res}}$ to ${\mathbb{R}}^m$. In the real world, such a functional is a machine or a device that processes an input v for a period $\left[-t,0\right]$ in real time and outputs its state at time 0.

For example, we define a functional using the following state-space system:

$$\dot{x}\left(t\right)=\varphi \left(x\left(t\right),u\left(t\right)\right),x\left(0\right)=x_{\mathrm{init}}\left(u:{\mathbb{R}}_{+}\to {\mathbb{R}}^n,t\ge 0\right),$$

(3)

where $x\left(t\right)\in {\mathbb{R}}^m$ and $u\left(t\right)\in {\mathbb{R}}^n$ are the system state and input at time $t\in {\mathbb{R}}_{+}=\left[0,\infty \right)$. The initial state is $x_{\mathrm{init}}\in {\mathbb{R}}^m$, and the derivative of the state is given by the function $\varphi$ of the state and input. In considering the fixed $x_{\mathrm{init}}$, System (3) determines $x\left(t\right)$ for $u\left(\tau \right)$$\left(\tau \in \left[0,t\right]\right)$. Hence, we can define the functional $f:V^{\mathrm{res}}\to {\mathbb{R}}^m$ as

$$f\left({v|}_{\left[-t,0\right]}\right)=x\left(0\right),\dot{x}\left(\tau \right)=\varphi \left(x\left(\tau \right),v\left(\tau \right)\right),x\left(-t\right)=x_{\mathrm{init}}\left(v\in V,t\ge 0,\tau \in \left[-t,0\right]\right).$$

(4)

Note that time is shifted so that the input signal starts at time $-t$ and ends at time 0. The functional output $f\left({v|}_{\left[-t,0\right]}\right)$ is the state to which the system transitions from the initial state $x_{\mathrm{init}}\in {\mathbb{R}}^m$ with the input ${v|}_{\left[-t,0\right]}$ of the length t. State spaces can represent many physical systems, and state-space systems can also be represented as functionals, as in the example. Therefore, our discussion of functionals covers a fairly wide range of dynamical systems.

In supervised learning, reservoir computing (RC) trains a model to approximate a functional we call the “target.” Let $f^{*}$ and $\widehat{f}:V^{\mathrm{res}}\to \mathbb{R}$ be the target and the model, respectively. We consider the uniform approximation, which is evaluated based on the worst error, ${sup}_{v\in V^{\mathrm{res}}}\left|f^{*}\left(v\right)-\widehat{f}\left(v\right)\right|$. The RC model is represented as the composition of two maps as $\widehat{f}=p\circ \mathit{f}$. The map $\mathit{f}:V^{\mathrm{res}}\to {\mathbb{R}}^m$ is the dynamical part of the model called the “reservoir,” and the map $p:{\mathbb{R}}^m\to \mathbb{R}$ is the static part called the “readout.” To approximate the target, RC trains only the readout and fixes the reservoir because training the dynamical part of the model is technically difficult and computationally expensive. Because the reservoir is fixed, we can implement it not only as software on a general-purpose computer but also as hardware. In the field of physical reservoir computing, many physical phenomena are studied as reservoirs and expected in terms of computation speed and energy consumption.

Let ${\mathcal{F}}^{*}$ be a set of targets, which are functionals from $V^{\mathrm{res}}$ to $\mathbb{R}$. An RC model and its reservoir are said to be universal in ${\mathcal{F}}^{*}$ if the model can approximate an arbitrary $f^{*}\in {\mathcal{F}}^{*}$. Let ${\mathbb{P}}_m$ be the set of polynomial functions from ${\mathbb{R}}^m$ to $\mathbb{R}$. If we use a polynomial readout, training the model involves selecting a polynomial function from ${\mathbb{P}}_m$, and universality in ${\mathcal{F}}^{*}$ is defined as follows.

Definition 1. 

Reservoir $\mathit{f}:V^{\mathrm{res}}\to {\mathbb{R}}^m$ is said to be universal for uniform approximations in ${\mathcal{F}}^{*}$ if

$$\forall f^{*}\in {\mathcal{F}}^{*},\forall \epsilon >0,\exists p\in {\mathbb{P}}_m,\forall v\in V^{\mathrm{res}},\left|f^{*}\left(v\right)-p\circ \mathit{f}\left(v\right)\right|<\epsilon .$$

(5)

Polynomial readouts are not practical because of their limited generalization ability. However, they are theoretically tractable, and the discussion on them can be extended to other types of readouts that can approximate continuous functions. Hence, it suffices to discuss the theoretical aspects by assuming polynomial readouts, even if we use other types of continuous functions, such as feed-forward neural networks. The definition of universality using a polynomial readout and uniform approximation, like Definition 1, has been discussed since the earliest studies on reservoir computing [3].

In previous studies, two sufficient conditions for universality were provided. One condition [20] is the weakest ever known and has been shown to be satisfiable using the other [21]. To explain these conditions, we need a metric on $V^{\mathrm{res}}\cup V$. Let $w:{\mathbb{R}}_{+}\to \left(0,1\right]$ be a non-increasing function that satisfies ${lim}_{t\to \infty }w\left(t\right)=0$. Using the supremum on the domain, we define the weighted norm ${∥v∥}_w$ of the function v as

$${∥v∥}_w=\left\{\begin{array}{cc}\underset{\tau \in \left[-t,0\right]}{sup}∥v\left(\tau \right)∥w\left(-\tau \right)\hfill & \left(v:\left[-t,0\right]\to {\mathbb{R}}^n\right)\hfill \\ \underset{\tau \le 0}{sup}∥v\left(\tau \right)∥w\left(-\tau \right)\hfill & \left(v:{\mathbb{R}}_{-}\to {\mathbb{R}}^n\right)\hfill \end{array}\right.,$$

(6)

where $t\in {\mathbb{R}}_{+}$. Let ${v|}_{\left[-\infty ,0\right]}=v$ for any $v\in V$ and define the map $\lambda :V^{\mathrm{res}}\cup V\to \left[0,\infty \right]$ as

$$\lambda \left({v|}_{\left[-t,0\right]}\right)=t\left(v\in V,t\in \left[0,\infty \right]\right).$$

(7)

For an input function, the map $\lambda$ returns the length of the interval on which that function is defined.

Let $\theta :{\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ be a strictly increasing, bounded, and continuous function. Using the weighted norm ${∥·∥}_w$ and function $\theta$, we define the map $d\left(v_1,v_2\right)$ as

$$d\left(v_1,v_2\right)={∥v_1{{|}_{\left[-t_{min},0\right]}-v_2|}_{\left[-t_{min},0\right]}∥}_w+\left|\theta \left(t_1\right)-\theta \left(t_2\right)\right|\left(v_1,v_2\in V^{\mathrm{res}}\cup V\right),$$

(8)

where $t_1=\lambda \left(v_1\right)$, $t_2=\lambda \left(v_2\right)$, $t_{min}=min\left\{t_1,t_2\right\}$, and $\theta \left(\infty \right)={lim}_{t\to \infty }\theta \left(t\right)$. The map d is a metric on $V^{\mathrm{res}}\cup V$ under the conditions we describe later. The first term of the distance (8) compares the inputs on the intersection of their domains. The function w assigns greater weight to the difference in the newer part of the inputs. The second term of (8) compares the length of the two inputs via the function $\theta$. From the definition of $\theta$, the second term is negligible if $t_1$ and $t_2$ are sufficiently large. The following proposition [20] provides the condition that makes d a metric:

Proposition 1. 

Suppose that the following triangle inequality holds for any $v_1$, $v_2\in V$, $t_1\ge 0$, and $t_2\ge t_1$:

$$d\left(v_1{{|}_{\left[-t_2,0\right]},v_2|}_{\left[-t_2,0\right]}\right)\le d\left(v_1{{|}_{\left[-t_1,0\right]},v_2|}_{\left[-t_2,0\right]}\right)+d\left(v_1{{|}_{\left[-t_1,0\right]},v_1|}_{\left[-t_2,0\right]}\right).$$

(9)

Then, $\left(V^{\mathrm{res}}\cup V,d\right)$ is a compact metric space, and $V^{\mathrm{res}}$ is dense in $V^{\mathrm{res}}\cup V$.

The triangle inequality (9) guarantees that d satisfies other triangle inequalities. The density of $V^{\mathrm{res}}$ is confirmed as follows: for any $v\in V$, ${v|}_{\left[-t,0\right]}\in V^{\mathrm{res}}$ converges to $v\in V$ as $t\to \infty$. From the density of $V^{\mathrm{res}}$, we have $V^{\mathrm{res}}\cup V=\overline{V^{\mathrm{res}}}$, where the symbol $\overline{·}$ is the closure. An example of a pair, $\left(w,\theta \right)$, that makes d a metric is shown by (38) in [20].

In the rest of this paper, we assume that targets in ${\mathcal{F}}^{*}$ are uniformly continuous following the previous study [3]. This assumption enables us to extended a functional $f^{*}\in {\mathcal{F}}^{*}$ onto $\overline{V^{\mathrm{res}}}$, i.e., there is a continuous functional ${\overline{f}}^{*}:\overline{V^{\mathrm{res}}}\to \mathbb{R}$ such that $f^{*}\left(v\right)={\overline{f}}^{*}\left(v\right)$ for any $v\in V^{\mathrm{res}}$. Such continuity on a compact set is needed for theoretical discussion on approximation.

The known weakest sufficient condition for universality is called the neighborhood separation property (NSP) [20]. The NSP means that the reservoir transfers the neighborhoods of distinct points to the images disconnected from each other. For $\delta >0$ and $v\in \overline{V^{\mathrm{res}}}$, we define the set $N_{\delta }\left(v\right)\subset V^{\mathrm{res}}$ as

$$N_{\delta }\left(v\right)=\left\{v^{\prime }\in V^{\mathrm{res}}\left|d\left(v,v^{\prime }\right)<\delta \right.\right\}.$$

(10)

Although the set $N_{\delta }\left(v\right)$ is similar to a general neighborhood, note that $N_{\delta }\left(v\right)\subset V^{\mathrm{res}}$ holds even if $v\in V$. The mathematical definition of the NSP of the reservoir $\mathit{f}:V^{\mathrm{res}}\to {\mathbb{R}}^m$ is the following.

Condition 1. 

For any distinct $v_1$, $v_2\in \overline{V^{\mathrm{res}}}$, some $\delta >0$ satisfies $\overline{\mathit{f}\left(N_{\delta }\left(v_1\right)\right)}\cap \overline{\mathit{f}\left(N_{\delta }\left(v_2\right)\right)}=\varnothing .$

Another sufficient condition for universality is that the reservoir $\mathit{f}$ has a uniformly continuous inverse, as shown below [21].

Condition 2. 

There is a uniformly continuous map, $g:\mathit{f}\left(V^{\mathrm{res}}\right)\to V^{\mathrm{res}}$, that satisfies $g\circ \mathit{f}={\mathrm{id}}_{V^{\mathrm{res}}}$.

Condition 2 means that the input function can be continuously reconstructed from the reservoir output and that there is a continuous map from the reservoir outputs to the target outputs through input functions. Condition 2 is sufficient for Condition 1, which shows that there is a universal reservoir. The Hahn–Mazurkiewicz theorem [25] claims that there is a continuous surjection $g:\left[0,1\right]\to \overline{V^{\mathrm{res}}}$, called a space-filling curve. Hence, we can obtain the reservoir $\mathit{f}:V^{\mathrm{res}}\to \mathbb{R}$, satisfying Condition 2, by restricting the domain of g for its image to be $V^{\mathrm{res}}$ and taking a right inverse of the restriction. This result can be easily extended to $\mathit{f}:V^{\mathrm{res}}\to {\mathbb{R}}^m$ with a general $m\in \mathbb{N}$.

3. Necessary and Sufficient Condition for Universality

In this section, we prove the following theorem:

Theorem 1. 

Let ${\mathcal{F}}^{*}$ be the set of uniformly continuous functionals from $V^{\mathrm{res}}$ to $\mathbb{R}$. Let $\mathit{f}:V^{\mathrm{res}}\to {\mathbb{R}}^m$ be a bounded functional. Then, Equation (5), Condition 1, and Condition 2 are equivalent to one another.

Theorem 1 means that a reservoir’s universality, NSP, and uniformly continuous inverse are equivalent. This is the first result that provides the necessary and sufficient conditions of a reservoir to be universal. Theorem 1 is proved as a corollary of the following generalization:

Theorem 2. 

Let X be a metric space with the metric d and $\overline{X}$ be a compactification of X. Suppose that $m\in \mathbb{N}$ and that $\mathit{f}:X\to {\mathbb{R}}^m$ is bounded. Then, the following three conditions are equivalent to one another:

(i)

For any uniformly continuous $f^{*}:X\to \mathbb{R}$ and $\epsilon >0$, there is some polynomial function, $p:{\mathbb{R}}^m\to \mathbb{R}$, that satisfies

$$\underset{x\in X}{sup}\left|f^{*}\left(x\right)-p\circ \mathit{f}\left(x\right)\right|<\epsilon .$$

(11)

(ii)

For any distinct $x_1$ and $x_2\in \overline{X}$, there is some $\delta >0$ that satisfies

$$\overline{\mathit{f}\left(N_{\delta }\left(x_1\right)\right)}\cap \overline{\mathit{f}\left(N_{\delta }\left(x_2\right)\right)}=\varnothing ,$$

(12)

where $N_{\delta }\left(x\right)=\left\{x^{\prime }\in X\left|d\left(x,x^{\prime }\right)<\delta \right.\right\}$ for $x\in \overline{X}$.

(iii)

There is a uniformly continuous map $g:\mathit{f}\left(X\right)\to X$ that satisfies

$$g\circ \mathit{f}={\mathrm{id}}_X,$$

(13)

where ${\mathrm{id}}_X$ is the identity mapping on X.

From Proposition 1, Theorem 1 is a corollary of Theorem 2 where $X=V^{\mathrm{res}}$. Conditions (i)–(iii) correspond to the universality of Definition 1 and Conditions 1 and 2, respectively. Set X does not have to be a strict subset of $\overline{X}$, i.e., $X=\overline{X}$ is allowed. Hence, instead of $V^{\mathrm{res}}$, we can consider the compact set V or $V\cup V^{\mathrm{res}}$ as X. In this case, we obtain the same result as that of Theorem 1 for input functions on an infinite time interval. We prove Theorem 2 by proving the following propositions: (iii)⇒(i), ¬(iii)$\Rightarrow \neg$(i), and ¬(iii)$\iff \neg$(ii). To prove propositions premised on ¬(iii), we use the following lemma:

Lemma 1. 

Suppose that Condition (iii) does not hold on the same premise as Theorem 2. Then, there exist the Cauchy sequences ${\left(x_{1,k}\right)}_{k\in \mathbb{N}}$ and ${\left(x_{2,k}\right)}_{k\in \mathbb{N}}\in X$ that satisfy

$$\underset{k\to \infty }{lim}{x}_{1,k}\ne \underset{k\to \infty }{lim}{x}_{2,k},\underset{k\to \infty }{lim}\mathit{f}\left(x_{1,k}\right)=\underset{k\to \infty }{lim}\mathit{f}\left(x_{2,k}\right).$$

(14)

Because of $x_{1,k}=g\circ \mathit{f}\left(x_{1,k}\right)$ and $x_{2,k}=g\circ \mathit{f}\left(x_{2,k}\right)$, Equation (14) means that even if two inputs to g are close, the outputs are not necessarily close. More precisely, g is not uniformly continuous if (14) holds. Lemma 1 claims the inverse of this proposition.

Proof of Lemma 1. 

First, we consider the case where no map $g:\mathit{f}\left(X\right)\to X$ satisfies $g\circ \mathit{f}={\mathrm{id}}_X$, i.e., $\mathit{f}$ does not have a left inverse. Having no left inverse is equivalent to not being injective. Hence, there exist distinct $x_1$ and $x_2\in X$ that satisfy $\mathit{f}\left(x_1\right)=\mathit{f}\left(x_2\right)$, and the sequences $x_{1,k}=x_1$ and $x_{2,k}=x_2$$\left(k\in \mathbb{N}\right)$, which do not change with k, satisfy (14).

Next, we consider the case where a map $g:\mathit{f}\left(X\right)\to X$ satisfying $g\circ \mathit{f}={\mathrm{id}}_X$ exists but is not uniformly continuous, i.e.,

$$\exists \epsilon >0,\forall \delta >0,\exists {\alpha }_1,{\alpha }_2\in \mathit{f}\left(X\right),∥{\alpha }_1-{\alpha }_2∥<\delta ,d\left(g\left({\alpha }_1\right),g\left({\alpha }_2\right)\right)\ge \epsilon .$$

(15)

Then, there exist $\epsilon >0$ and the sequences ${\left({\alpha }_{1,k}\right)}_{k\in \mathbb{N}}$, ${\left({\alpha }_{2,k}\right)}_{k\in \mathbb{N}}\in \mathit{f}\left(X\right)$ that satisfy

$$∥{\alpha }_{1,k}-{\alpha }_{2,k}∥<{\displaystyle \frac{1}{k}},d\left(g\left({\alpha }_{1,k}\right),g\left({\alpha }_{2,k}\right)\right)\ge \epsilon \left(k\in \mathbb{N}\right).$$

(16)

A bounded $\mathit{f}$ means that $\overline{\mathit{f}\left(X\right)}\subset {\mathbb{R}}^m$ is bounded and closed, i.e., compact. An infinite sequence on a compact metric space includes a subsequence that converges on that space. Hence, there exist an infinite set ${\mathbb{N}}^{\prime }\subset \mathbb{N}$ and $\alpha \in \overline{\mathit{f}\left(X\right)}$ such that the subsequence ${\left({\alpha }_{1,k}\right)}_{k\in {\mathbb{N}}^{\prime }}$ converges to $\alpha$. From the first inequality in (16), we have

$${\alpha }_{1,k}\to \alpha ,{\alpha }_{2,k}\to \alpha \left(k\to \infty ,k\in {\mathbb{N}}^{\prime }\right).$$

(17)

Because ${\alpha }_{1,k}$ and ${\alpha }_{2,k}$ are included in $\mathit{f}\left(X\right)$, there exist the sequences ${\left(x_{1,k}\right)}_{k\in \mathbb{N}}$ and ${\left(x_{2,k}\right)}_{k\in \mathbb{N}}\in X$ that satisfy the following:

$$\mathit{f}\left(x_{1,k}\right)={\alpha }_{1,k},\mathit{f}\left(x_{1,k}\right)={\alpha }_{2,k}\left(k\in \mathbb{N}\right).$$

(18)

We show that ${\left(x_{1,k}\right)}_{k\in \mathbb{N}}$ and ${\left(x_{2,k}\right)}_{k\in \mathbb{N}}$ include subsequences that satisfy (14). From (17) and (18), the subsequences ${\left(x_{1,k}\right)}_{k\in {\mathbb{N}}^{\prime }}$ and ${\left(x_{2,k}\right)}_{k\in {\mathbb{N}}^{\prime }}$ satisfy the second equation in (14). Because $\overline{X}$ is compact, there is some infinite set, ${\mathbb{N}}^{″}\subset {\mathbb{N}}^{\prime }$, such that the subsequence ${\left(x_{1,k}\right)}_{k\in {\mathbb{N}}^{″}}$ converges. Similarly, there is also some infinite set, ${\mathbb{N}}^{‴}\subset {\mathbb{N}}^{″}$, such that the subsequence ${\left(x_{2,k}\right)}_{k\in {\mathbb{N}}^{‴}}$ converges. Using g in (18) and substituting $g\circ \mathit{f}={\mathrm{id}}_X$, the following holds:

$$x_{1,k}=g\left({\alpha }_{1,k}\right),x_{2,k}=g\left({\alpha }_{2,k}\right)\left(k\in \mathbb{N}\right).$$

(19)

From (19) and the second inequality in (16), we have $d\left(x_{1,k},x_{2,k}\right)\ge \epsilon$ for any $k\in \mathbb{N}$. Therefore, the Cauchy sequences ${\left(x_{1,k}\right)}_{k\in {\mathbb{N}}^{‴}}$ and ${\left(x_{2,k}\right)}_{k\in {\mathbb{N}}^{‴}}$ satisfy the first equation in (14), which proves Lemma 1. □

Theorem 2 is proved as follows:

Proof of Theorem 2. 

As we explained, we prove Propositions (iii)⇒(i), ¬(iii)$\Rightarrow \neg$(i), ¬(iii)$\Rightarrow \neg$(ii), and ¬(ii)$\Rightarrow \neg$(iii).

Proof of (iii)⇒(i): Let $f^{*}:X\to \mathbb{R}$ and $\epsilon >0$ be an arbitrary uniformly continuous functional and an arbitrary real number. We define a function $q:\mathit{f}\left(X\right)\to \mathbb{R}$ as $q=f^{*}\circ g$. Because $f^{*}=f^{*}\circ g\circ \mathit{f}=q\circ \mathit{f}$, (11) is equivalent to

$$\underset{x\in X}{sup}\left|q\circ \mathit{f}\left(x\right)-p\circ \mathit{f}\left(x\right)\right|<\epsilon .$$

(20)

The function q is uniformly continuous because $f^{*}$ and g are uniformly continuous. Hence, there is a unique continuous extension, $\overline{q}:\overline{\mathit{f}\left(X\right)}\to \mathbb{R}$, satisfying $\overline{q}\left(\alpha \right)=q\left(\alpha \right)$ for any $\alpha \in \mathit{f}\left(X\right)$. From the Stone–Weierstrass theorem [26], there is some polynomial function $p:{\mathbb{R}}^m\to \mathbb{R}$ that satisfies $\left|\overline{q}\left(\alpha \right)-p\left(\alpha \right)\right|<\epsilon$ for any $\alpha$ on the compact set $\overline{\mathit{f}\left(X\right)}$. The polynomial function p also satisfies (20), which proves (iii)⇒(i). The relationship among maps in the proof is shown in Figure 3.

Proof of ¬(iii)$\Rightarrow \neg$(i): From Lemma 1, there exist the Cauchy sequences ${\left(x_{1,k}\right)}_{k\in \mathbb{N}}$ and ${\left(x_{2,k}\right)}_{k\in \mathbb{N}}\in X$ that satisfy (14). We define distinct $x_1$ and $x_2\in \overline{X}$ as

$$x_1=\underset{k\to \infty }{lim}{x}_{1,k},x_2=\underset{k\to \infty }{lim}{x}_{2,k}.$$

(21)

Let $f^{*}:X\to \mathbb{R}$ be a uniformly continuous function that satisfies

$$\underset{k\to \infty }{lim}{f}^{*}\left(x_{1,k}\right)=0,\underset{k\to \infty }{lim}{f}^{*}\left(x_{2,k}\right)=1.$$

(22)

A specific definition of $f^{*}$ is not necessary for the proof, but it can be defined as follows:

$$f^{*}\left(x\right)={\displaystyle \frac{d\left(x_1,x\right)}{d\left(x_1,x_2\right)}}\left(x\in X\right).$$

(23)

To prove by contradiction, we suppose that (i) holds and let $\epsilon =1/3$. Then, there is some polynomial function $p:{\mathbb{R}}^m\to \mathbb{R}$ that satisfies

$$\left|f^{*}\left(x_{1,k}\right)-p\circ \mathit{f}\left(x_{1,k}\right)\right|<{\displaystyle \frac{1}{3}},\left|f^{*}\left(x_{2,k}\right)-p\circ \mathit{f}\left(x_{2,k}\right)\right|<{\displaystyle \frac{1}{3}}\left(k\in \mathbb{N}\right).$$

(24)

From (14) and the continuity of p, the following holds:

$$\underset{k\to \infty }{lim}p\circ \mathit{f}\left(x_{1,k}\right)=\underset{k\to \infty }{lim}p\circ \mathit{f}\left(x_{2,k}\right).$$

(25)

However, from (22) and the limit of (24) when $k\to \infty$, the following also holds:

$$\underset{k\to \infty }{lim}p\circ \mathit{f}\left(x_{1,k}\right)<{\displaystyle \frac{1}{3}},{\displaystyle \frac{2}{3}}<\underset{k\to \infty }{lim}p\circ \mathit{f}\left(x_{1,k}\right).$$

(26)

This contradicts (25), and we have ¬(i). The relationship among sets and sequences in the proof is shown in Figure 4.

Proof of ¬(iii)$\Rightarrow \neg$(ii): From Lemma 1, there exist the Cauchy sequences ${\left(x_{1,k}\right)}_{k\in \mathbb{N}}$ and ${\left(x_{2,k}\right)}_{k\in \mathbb{N}}\in X$ that satisfy (14). We define distinct $x_1$ and $x_2\in \overline{X}$ and $\alpha \in \overline{\mathit{f}\left(X\right)}$ as

$$x_1=\underset{k\to \infty }{lim}{x}_{1,k},x_2=\underset{k\to \infty }{lim}{x}_{2,k},\alpha =\underset{k\to \infty }{lim}\mathit{f}\left(x_{1,k}\right)=\underset{k\to \infty }{lim}\mathit{f}\left(x_{2,k}\right).$$

(27)

Let $\delta >0$ be an arbitrary real number. From the first and second equations in (27), there is a sufficiently large $k\in \mathbb{N}$ that satisfies $\mathit{f}\left(x_{1,k}\right)\in \mathit{f}\left(N_{\delta }\left(x_1\right)\right)$ and $\mathit{f}\left(x_{2,k}\right)\in \mathit{f}\left(N_{\delta }\left(x_2\right)\right)$. Because both $\mathit{f}\left(x_{1,k}\right)$ and $\mathit{f}\left(x_{2,k}\right)$ converge to $\alpha$ as $k\to \infty$, we have $\alpha \in \overline{\mathit{f}\left(N_{\delta }\left(x_1\right)\right)}\cap \overline{\mathit{f}\left(N_{\delta }\left(x_2\right)\right)}$, which proves ¬(iii)$\Rightarrow \neg$(ii). The relationship among sets and sequences in the proof is shown in Figure 5.

Proof of ¬(ii)$\Rightarrow \neg$(iii): Suppose that distinct $x_1$ and $x_2\in \overline{X}$ satisfy $\overline{\mathit{f}\left(N_{\delta }\left(x_1\right)\right)}\cap \overline{\mathit{f}\left(N_{\delta }\left(x_2\right)\right)}\ne \varnothing$ for any $\delta >0$. Let $\delta =d\left(x_1,x_2\right)/3$. We consider the sequences ${\left({\alpha }_{1,k}\right)}_{k\in \mathbb{N}}\in \mathit{f}\left(N_{\delta }\left(x_1\right)\right)$ and ${\left({\alpha }_{2,k}\right)}_{k\in \mathbb{N}}\in \mathit{f}\left(N_{\delta }\left(x_2\right)\right)$ that converge to $\alpha \in \overline{\mathit{f}\left(N_{\delta }\left(x_1\right)\right)}\cap \overline{\mathit{f}\left(N_{\delta }\left(x_2\right)\right)}$. To prove by contradiction, suppose that a uniformly continuous map $g:\mathit{f}\left(X\right)\to X$ exists and satisfies $g\circ \mathit{f}={\mathrm{id}}_X$. Then, we have $g\left({\alpha }_{1,k}\right)\in N_{\delta }\left(x_1\right)$ and $g\left({\alpha }_{2,k}\right)\in N_{\delta }\left(x_2\right)$. Therefore, although ${\alpha }_{1,k}$ and ${\alpha }_{2,k}$ converge to the same point $\alpha$ as $k\to \infty$, we have $d\left(g\left({\alpha }_{1,k}\right),g\left({\alpha }_{2,k}\right)\right)>d\left(x_1,x_2\right)/3$ for any $k\in \mathbb{N}$. This means that g is not uniformly continuous, and we have ¬(iii). The relationship among sets and sequences in the proof is shown in Figure 6.

From the above four propositions, we have (iii)⇔(i) and (iii)⇔(ii), which proves Theorem 2. □

4. Pathological Property of Universal Reservoir

Although a previous study [21] showed that a universal reservoir exists mathematically, whether we can physically construct one is still unknown. The authors of [21] suggested that the proposed universal reservoir using a space-filling curve has an infinite number of discontinuous points and is difficult to implement. In this section, we show that all universal reservoirs have the same problem as the example in [21]. The main result of this section is the following:

Theorem 3. 

Let ${\mathcal{F}}^{*}$ be the set of uniformly continuous functionals from $V^{\mathrm{res}}$ to $\mathbb{R}$. Let $\mathit{f}:V^{\mathrm{res}}\to {\mathbb{R}}^m$ be a functional that is bounded and universal for uniform approximations in ${\mathcal{F}}^{*}$. Then, the set of points $v\in V^{\mathrm{res}}$ at which $\mathit{f}$ is discontinuous is dense in $V^{\mathrm{res}}$.

Theorem 3 says that a universal reservoir has a discontinuous point on an arbitrary neighborhood of an arbitrary point and is very sensitive to input change. Note that the discontinuity of $f:V^{\mathrm{res}}\to \mathbb{R}$ does not mean the discontinuity of the time derivative $\varphi$ of states in the state-space representation (4) of f. The same holds for $\mathit{f}$, which has m outputs. Theorem 3 is proved using the contradiction that if $\mathit{f}$ is continuous on some open set, two spaces with different dimensions are homeomorphic. To this end, we use the following lemma about a topological embedding:

Lemma 2. 

For any $v\in V^{\mathrm{res}}$, $\delta >0$, and any $l\in \mathbb{N}$, there is some topological embedding $h:{\left[0,1\right]}^l\to N_{\delta }\left(v\right)$.

Proof of Lemma 2. 

Let $v\in V^{\mathrm{res}}$ be an arbitrary function. Let $\delta >0$ and l be arbitrary real and natural numbers, respectively. Let $t=\lambda \left(v\right)$, i.e., the domain of v is $\left[-t,0\right]$. Because the bounded function $\theta :{\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ in (8) is continuous and monotonically increasing, there is some $T>0$ satisfying $\theta \left(t+lT\right)-\theta \left(t\right)<\delta$. Let an input value $a\in A$ satisfy $0<∥a-v\left(-t\right)∥\le KT$, where A is the input value space and K is the maximum Lipschitz constant in (1). Using T and a, we define the map $h:{\left[0,1\right]}^l\to N_{\delta }\left(v\right)$ as

$$h\left(\mathit{\beta }\right)=v_{\mathit{\beta }},v_{\mathit{\beta }}\left(\tau \right)=\left\{\begin{array}{cc}v\left(\tau \right)\hfill & \left(-t<\tau \le 0\right),\hfill \\ v\left(-t\right)+\left[a-v\left(-t\right)\right]r_{\mathit{\beta }}\left(\tau \right)\hfill & \left(-t-lT\le \tau \le -t\right)\hfill \end{array}\right.\left(\mathit{\beta }\in {\left[0,1\right]}^l\right),$$

(28)

where $r_{\beta }:\left[-lT-t,-t\right]\to \left[0,1\right]$ is defined for $\mathit{\beta }=\left[{\beta }_1\dots {\beta }_l\right]$ as

$$r_{\mathit{\beta }}\left(\tau \right)=\sum _{i=1}^l{\beta }_{i}max\left\{1-{\displaystyle \frac{\left|\tau +iT+t\right|}{T}},0\right\}\left(-t-lT\le \tau \le -t\right).$$

(29)

As shown in Figure 7, $v_{\mathit{\beta }}$ is an extension of v to $\left[-t-lT,0\right]$ with the values on the added domain defined as some internal division point between $v\left(-t\right)$ and a. The division rate $r_{\mathit{\beta }}$ is piece-wise linear and takes the value of ${\beta }_i$ at $-t-iT$ for $i\in \left\{1,\dots ,l\right\}$, which are the borders of the linear pieces.

First, we show that $v_{\mathit{\beta }}\in N_{\delta }\left(v\right)$ holds for any $\mathit{\beta }\in {\left[0,1\right]}^l$. Because the convex set A includes a and $v\left(-t\right)$, the image of $v_{\mathit{\beta }}$ is included in A. Because the Lipschitz constant of $r_{\mathit{\beta }}$ is at most $1/T$, and $∥a-v\left(-t\right)∥\le KT$ holds, the Lipschitz constant of $v_{\mathit{\beta }}$ is less than or equal to K. Hence, we have $v_{\mathit{\beta }}\in V^{\mathrm{res}}$. The functions v and $v_{\mathit{\beta }}$ have the same values on the intersection of their domain. Hence, from the definition of T, we have $d\left(v,v_{\mathit{\beta }}\right)=\theta \left(t+lT\right)-\theta \left(t\right)<\delta$, i.e., $v_{\mathit{\beta }}\in N_{\delta }\left(v\right)$.

Next, we show that h is an embedding. A continuous bijection from a compact set is homeomorphic, and the domain ${\left[0,1\right]}^l$ of h is compact. Hence, if h is a continuous injection, it is homeomorphic to its image, i.e., an embedding. As we explained, we have $r_{\mathit{\beta }}\left(-t-iT\right)={\beta }_i$ for $i\in \left\{1,\dots ,l\right\}$. Hence, different $\mathit{\beta }$ gives different $r_{\mathit{\beta }}$ and $h\left(\mathit{\beta }\right)$, i.e., the map h is injective. Let $v_{{\mathit{\beta }}_1}=h\left({\mathit{\beta }}_1\right)$ and $v_{{\mathit{\beta }}_2}=h\left({\mathit{\beta }}_2\right)$ for ${\mathit{\beta }}_1$, ${\mathit{\beta }}_2\in {\left[0,1\right]}^l$. Then, the distance between $v_{{\mathit{\beta }}_1}$ and $v_{{\mathit{\beta }}_2}$ is written as

$$d\left(v_{{\mathit{\beta }}_1},v_{{\mathit{\beta }}_2}\right)=\underset{-t-lT\le \tau \le 0}{sup}∥v_{{\mathit{\beta }}_1}\left(\tau \right)-v_{{\mathit{\beta }}_2}\left(\tau \right)∥\le ∥a-v\left(-t\right)∥∥{\mathit{\beta }}_1-{\mathit{\beta }}_2∥.$$

(30)

This means that the map h is continuous. Therefore, the map $h:{\left[0,1\right]}^l\to N_{\delta }\left(v\right)$ is an embedding, which proves Lemma 2. □

Using Lemma 2, we prove Theorem 3 as follows:

Proof of Theorem 3. 

Let bounded $\mathit{f}:V^{\mathrm{res}}\to {\mathbb{R}}^m$ be universal for uniform approximations in ${\mathcal{F}}^{*}$. To prove by contradiction, suppose that discontinuous points of $\mathit{f}$ are not dense in $V^{\mathrm{res}}$, i.e., there exist $v\in V^{\mathrm{res}}$ and $\delta >0$ such that $\mathit{f}$ is continuous on $N_{\delta }\left(v\right)$. From Theorem 1, universality is equivalent to Condition 2, and $\mathit{f}$ has a continuous inverse map. Hence, the restriction ${\mathit{f}|}_{N_{\delta }\left(v\right)}$ of $\mathit{f}$ to $N_{\delta }\left(v\right)$ is a topological embedding because it and its inverse are continuous. From Lemma 2, there is a topological embedding $h:{\left[0,1\right]}^{m+1}\to N_{\delta }\left(v\right)$. The composition ${\mathit{f}|}_{N_{\delta }\left(v\right)}\circ h$ is also an embedding, i.e., ${\left[0,1\right]}^{m+1}$ and ${\mathit{f}|}_{N_{\delta }\left(v\right)}\circ h\left({\left[0,1\right]}^{m+1}\right)$ are homeomorphic. The relationship among the domains and images of h and $\mathit{f}$ is shown in Figure 8.

We call the small inductive dimension simply a dimension and write the dimension of the space A as $\mathrm{ind}A$. Dimensions have the following two properties (see pages 3–4 of [27]): First, two homeomorphic topological spaces have the same dimension. Second, a topological space has a dimension equal to its subspace or larger. Therefore, we have the contradiction of

$$m+1=\mathrm{ind}{\left[0,1\right]}^{m+1}{=\mathrm{ind}\mathit{f}|}_{N_{\delta }\left(v\right)}\circ h\left({\left[0,1\right]}^{m+1}\right)\le \mathrm{ind}{\mathbb{R}}^m=m,$$

(31)

which proves Theorem 3. □

5. Conclusions

We show that the universality of a reservoir and its two sufficient conditions, the NSP and the continuous inverse, are equivalent to one another. We also show that a reservoir has dense discontinuous points if it has a continuous inverse. Our results indicate that universality, discussed since [3], is an ideal approximation ability difficult to achieve in practice.

The behavior of a universal reservoir, as shown in this study, indicates that chaotic systems are necessary to achieve or approach universality. The difference between two inputs remains in a state-space system as the difference between states. Hence, a system sensitive to inputs is also sensitive to the initial state. As we showed, a universal reservoir has dense discontinuous points and is very sensitive to inputs. Therefore, the universality can be related to chaotic systems. Some studies support this interpretation [19,22,23,24].

Our results also provide an insight to guarantee a weaker but practical approximation ability. We reveal that a reservoir’s “resolution,” ability to distinguish inputs, is directly linked to the approximation ability of the RC model. The NSP can be seen as an infinite resolution that distinguishes two input functions, regardless of how similar they are. We can also consider reservoirs with finite resolution, and it may be possible to guarantee approximation abilities for them according to their resolution. Such reservoirs behave more gently and are easier to implement than those with the NSP.