Quantifying Emergent Behavior of Autonomous Robots

Abstract

Quantifying behaviors of robots which were generated autonomously from task-independent objective functions is an important prerequisite for objective comparisons of algorithms and movements of animals. The temporal sequence of such a behavior can be considered as a time series and hence complexity measures developed for time series are natural candidates for its quantification. The predictive information and the excess entropy are such complexity measures. They measure the amount of information the past contains about the future and thus quantify the nonrandom structure in the temporal sequence. However, when using these measures for systems with continuous states one has to deal with the fact that their values will depend on the resolution with which the systems states are observed. For deterministic systems both measures will diverge with increasing resolution. We therefore propose a new decomposition of the excess entropy in resolution dependent and resolution independent parts and discuss how they depend on the dimensionality of the dynamics, correlations and the noise level. For the practical estimation we propose to use estimates based on the correlation integral instead of the direct estimation of the mutual information based on next neighbor statistics because the latter allows less control of the scale dependencies. Using our algorithm we are able to show how autonomous learning generates behavior of increasing complexity with increasing learning duration.

1. Introduction

Whenever we create behavior in autonomous robots we strive for a suitable measure in order to quantify success, learning progress or compare algorithms with each other. When a specific task is given, then typically the task provides a natural measure of success, for instance walking may be measured by velocity and perturbation stability. In cases where behavior is learned via optimization of a global objective function then the same function can also be used as a quantification, so creating and quantifying behavior often go hand in hand. This also applies in principle to behavior from task-independent objectives that have been recently more and more successful in generating emergent autonomous behavior in robots [1,2,3,4,5,6,7]. A task is here understood as a specific desired target behavior such as walking or reaching.

However, there are several cases of emergent behavior where this strategy fails: if the behavior arises from (a) optimizing a local function [8,9], (b) optimizing a crude approximation of a computationally expensive objective function [9], (c) local interaction rules without an explicit optimization function [10,11,12], and (d) a biological system (e.g., freely moving animals) where we do not know the underlying optimization function. Thus, independent of the origin of behavior it would be useful to have a quantitative description of its structure. This would allow to objectively compare algorithms with each other and to compare technical with biological autonomous systems. This paper presents a measure of behavioral complexity that is suitable for the analysis of this kind of emergent behavior.

We base our measure on the predictive information (PI) [13] of a set of observables, such as joint angles. The PI is the mutual information between past and future of a time-series and captures how much information (in bits) can be used from the past to predict the future. It is also closely linked to excess entropy [14,15] or effective measure complexity [16], and it is a natural measure for the complexity of a time series because it provides a lower bound for the information necessary for optimal prediction. Thus it is a promising choice as a measure. Given these favorable properties, the PI was also proposed as an intrinsic drive for behavior generation in autonomous robots [1,2,9]. On an intuitive level maximizing the PI of the sensor process, leads to a high variety of sensory input—large marginal entropy—while keeping a temporal structure—high predictability corresponding to a small entropy rate.

Unfortunately, there are conceptual and practical challenges in using the PI for generating and measuring autonomous behavior. Conceptually, for systems with continuous values the PI provides a family of functions depending on the resolution of the measurements and practically it is difficult to estimate the PI. For a fixed partition (single resolution) in a low-dimensional case it is possible to estimate the one-step PI and adapt a controller to maximize it [2]. In high-dimensional systems it cannot be used directly in an online algorithm. An alternative approach [9] uses a dynamical systems model, linearization and a time-local version of the PI [9] to obtain an explicit gradient for locally maximizing PI for continuous and high-dimensional state spaces (no global optimization). A simplified version of the resulting neural controller [11] was used for generating the examples presented in this paper.

The aim of the present paper is to provide a measure of behavioral complexity that can be applied to emergent autonomous behavior, animal behavior and potentially other time-series. In order to do this it turns out that the predictive information as a complexity measure has to be refined by taking into account its scale dependency. Because the systems of interest are high-dimensional we investigate different methods to estimate information theoretic quantities in a resolution dependent way and present results for different robotic experiments. The emergent behavior in these experiments are generated by the above mentioned learning rules. On the one hand we want to increase our understanding of the learning process guided by the time-local version of the PI and on the other hand demonstrate how the resolution dependence allows us to quantify behavior on different length-scales.

We argue that many natural behaviors are low-dimensional, at least on a coarse scale. For instance in walking all joint angles can be typically related to a single phase, so in the extreme case it is one-dimensional [17]. However, as we show below in Section 2.1, this contradicts to the global maximization of the PI which would maximize the dimension. We discuss this contradiction and how it was circumvented in the practical applications. Along with the paper there is a supplementary material and source code, see [18,19].

The plan of the paper is as follows. In Section 2.1, we introduce the information theoretic quantities and in Section 2.3 the two estimation methods used in the paper. In Section 2.4 we describe the algorithm used to calculate the quantification measures. In Section 2.5 we demonstrate their behavior at the example of the Lorenz system including the effects of noise. Section 3 starts with explaining the robots and the control framework used for the experiments which are described in Section 3.2. The results of the experiments quantification are presented in Section 3.3 and are discussed in Section 4.

2. Methods

2.1. Entropy, Dimension and Excess Entropy

Starting point is a (in general) vector valued stationary time series $\{y_0,y_1,y_2,\cdots ,y_N\}$. It is used to reconstruct the phase space of the underlying dynamical system by using delay embedding: ${\overrightarrow{x}}_t=\{y_t,y_{t-\tau },y_{t-2\tau },\cdots ,y_{t-(m-1)\tau }\}$. If the original time series was $d-$dimensional the reconstructed phase space will be of dimension $d\times m$. More general embeddings are also possible, e.g., involving different delays or different embedding dimensions for each component of the time series. In the following, we will only consider the case $d=1$, i.e., scalar time series. The generalization to vector-values time series is straight forward. Let us assume that we are able to reconstruct approximately the joint probability distribution $p({\overrightarrow{x}}_t,{\overrightarrow{x}}_{t-1},...)$ in a reconstructed phase space. Then we can characterize its structure using measures from information theory. Information theoretic measures represent a general and powerful tool for the characterization of the structure of joint probability distributions [20,21]. The uncertainty about the outcome of a single measurement of the state, i.e., about ${\overrightarrow{x}}_t$ is given by its entropy. For discrete-valued random variable X with values $x\in \mathcal{X}$ and a probability distribution $p\left(x\right)$ it is defined as

$$H\left(X\right)=-\sum _{x\in \mathcal{X}}p\left(x\right)logp\left(x\right).$$

An alternative interpretation for the entropy is the average number of bits required to encode a new measurement. In our case, however, the $y_t$ are considered as continuous-valued observables (that are measured with a finite resolution). For continuous random variables with a probability density $\rho \left(x\right)$ one can also define an entropy, the differential entropy

$$H^C\left(X\right)=-\int \rho \left(x\right)log\rho \left(x\right)dx.$$

(1)

However, it behaves differently than its discrete counterpart: It can become negative and it will get even minus infinity if the probability measure for X is not absolutely continuous with respect to the Lebesgue measure—for instance, in the case of the invariant measure of a deterministic system with an attractor dimension smaller than the phase space dimension. Therefore, when using information theoretic quantities for characterizing dynamical systems researchers often prefer using the entropy for discrete-valued random variables. In order to use them for dynamical systems with continuous variables usually either partitions of the phase space or entropy-like quantities based on coverings are employed. These methods do not explicitly reconstruct the underlying invariant measure, but exploit the neighbor statistics directly. Alternatively one could use direct reconstructions using kernel density estimators [22] or methods based on maximizing relative entropy [23,24] to gain parametric estimates. These reconstructions, however, will always lead to probability densities, and are not suitable for reconstructing fractal measures which appear as invariant measures of deterministic systems.

In this paper, we use estimators based on coverings, i.e., correlation entropies [25] and nearest neighbors based methods [26] considered in Section 2.3 below. For the moment let us consider a partition of the phase space into hypercubes with side-length ϵ. For a more general definition of an ϵ-partition see [27]. In principle one might consider scaling the different dimensions of $y_t$ differently, but for the moment we assume that the time series was measured using an appropriate rescaling. The entropy of the state vector ${\overrightarrow{x}}_t$ observed with a ε-partition will be denoted in the following as $H(\overrightarrow{X};\epsilon )$ with ϵ parameterizing the resolution.

How does the entropy change if we change ε? The uncertainty about the potential outcome of a measurement will increase if the resolution of the measurement is increased, because of the larger number of potential outcomes. If $\overrightarrow{x}$ is a m-dimensional random variable and distributed according to a corresponding probability density function $\rho \left(x\right)$ we have asymptotically for $ϵ\to 0$ see ([27] or [28] (ch. 8, p. 248, theorem 8.3.1)).

$$H(X;\epsilon )\approx H^C\left(X\right)-mlog\epsilon .$$

(2)

This is what we would expect for a stochastic system. However, if we observe a deterministic system the behavior of an observable depends how its dimension relates to the attractor dimension. If the embedding dimension is smaller than the attractor dimension the deterministic character will not be resolved and Equation (2) still applies. However, if the embedding dimension is sufficiently high ($m>D$ [29]) then instead of a density function $\rho \left(x\right)$ we have to deal with a D-dimensional measure $\mu \left(x\right)$ and the entropy will behave as

$$H(X;\epsilon )\approx \mathrm{const}-Dlog\epsilon .$$

(3)

If an behavior such as in Equations (2) or (3) is observed for a range of ϵ values we will call this range a stochastic or deterministic scaling range, respectively.

Conditional entropy and mutual information Let us consider two discrete-valued random variables X and Y with values $x\in \mathcal{X}$ and $y\in \mathcal{Y}$, respectively. Then the uncertainty of a measurement of X is quantified by $H\left(X\right)$. Now we might ask, what is the average remaining uncertainty about X if we have seen already Y? This is quantified by the conditional entropy

$$H\left(X\right|Y)=H(X,Y)-H(Y).$$

(4)

The reduction of uncertainty about X knowing Y is the information that Y provides about X and is called the mutual information between X and Y

$$I(X:Y)=H\left(X\right)-H\left(X\right|Y).$$

(5)

Having defined the ε-dependent state entropy $H(\overrightarrow{X};\epsilon )$ we can now ask, how much information the present state contains about the state of the system at the next time step. The answer is given by the mutual information between ${\overrightarrow{X}}_t$ and ${\overrightarrow{X}}_{t+1}$:

$$I({\overrightarrow{X}}_t:{\overrightarrow{X}}_{t+1};\epsilon )=H({\overrightarrow{X}}_{t+1};\epsilon )-H\left({\overrightarrow{X}}_{t+1}\right|{\overrightarrow{X}}_t;\epsilon ).$$

(6)

Using Equation (2) one see, that for stochastic systems the mutual information will remain finite In the limit $\epsilon \to 0$ and can be expressed by the differential entropies:

$$I({\overrightarrow{X}}_t:{\overrightarrow{X}}_{t+1}):=\underset{\epsilon \to 0}{lim}I({\overrightarrow{X}}_t:{\overrightarrow{X}}_{t+1};\epsilon )=H^C\left({\overrightarrow{X}}_{t+1}\right)-H^C\left({\overrightarrow{X}}_{t+1}\right|{\overrightarrow{X}}_t)$$

(7)

Note that this mutual information is invariant with respect to coordinate transformation of the system state, i.e., if ${\overrightarrow{z}}_j=f\left(x_j\right)$ is a continuous and invertible function, then

$$I({\overrightarrow{X}}_t:{\overrightarrow{X}}_{t+1})=I({\overrightarrow{Z}}_t:{\overrightarrow{Z}}_{t+1}).$$

(8)

However, in the case of a deterministic system, the mutual information will diverge

$$I({\overrightarrow{X}}_t:{\overrightarrow{X}}_{t+1};\epsilon )\propto -Dlog\epsilon \text{for}\epsilon \to 0.$$

(9)

This is reasonable behavior because in principle the actual state contains an arbitrary large amount of information about the future. In practice, however, the state is known only with a finite resolution determined by the measurement device or the noise level.

Predictive information, excess entropy and entropy rate The unpredictability of a time series can be characterized by the conditional entropy of the next state given the previous states. In the following we will use an abbreviated notation for these conditional entropies and the involved entropies:

$$\begin{array}{cc}\hfill H_m({\overrightarrow{X}}_t;\epsilon )& :=H({\overrightarrow{X}}_t,{\overrightarrow{X}}_{t-1},\dots ,{\overrightarrow{X}}_{t-m+1};\epsilon )\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill h_m({\overrightarrow{X}}_t;\epsilon )& :=H\left({\overrightarrow{X}}_t\right|{\overrightarrow{X}}_{t-1},\dots ,{\overrightarrow{X}}_{t-m};\epsilon )\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill h_0({\overrightarrow{X}}_t;\epsilon )& :=H({\overrightarrow{X}}_t;\epsilon ).\hfill \end{array}$$

The entropy rate ([28] see (ch. 4.2)) is this conditional information if we condition on the infinite past

$$h_{\infty }({\overrightarrow{X}}_t;\epsilon )=\underset{m\to \infty }{lim}{h}_m({\overrightarrow{X}}_t;\epsilon )$$

(12)

In the following we assume stationarity, i.e., we have no explicit time dependence of the joint probabilities and therefore also of the entropies. Moreover, if it is clear from the context, which stochastic process is considered, we will write $H_m\left(\epsilon \right)$ and $h_m\left(\epsilon \right)$ instead of $H_m({\overrightarrow{X}}_t;\epsilon )$ and $h_m({\overrightarrow{X}}_t;\epsilon )$, respectively and it holds

$$h_m\left(\epsilon \right)=H_{m+1}\left(\epsilon \right)-H_m\left(\epsilon \right).$$

(13)

For deterministic systems the entropy rate will converge in the limit $\epsilon \to 0$ to the Kolmogorov-Sinai (KS-)entropy [30,31] which is a dynamical invariant of the system in the sense that it is independent on the specific state space reconstruction. Moreover, already for finite $m\ge D$, $h_m\left(\epsilon \right)$ will not depend on ε for sufficiently small ε because of Equation (3).

To quantify the amount of predictability in a state sequence one might consider subtracting the unpredictable part from the total entropy of a state sequence. By doing this one ends up with a well known complexity measure for time series, the excess entropy [14,15] or effective measure complexity [16]

$$E\left(\epsilon \right)=\underset{m\to \infty }{lim}{E}_m\left(\epsilon \right)$$

(14)

with

$$E_m\left(\epsilon \right):=H_m\left(\epsilon \right)-m{h}_{m-1}\left(\epsilon \right).$$

(15)

The excess entropy provides a lower bound for the amount of information necessary for an optimal prediction. For deterministic systems, however, it will diverge because $H_m\left(ϵ\right)$ will behave according to Equation (3) and $h_{m-1}\left(\epsilon \right)$ will become ϵ-independent for sufficiently large m and small ϵ, respectively

$$E_m\left(\epsilon \right)\approx \mathrm{const}-Dlog\epsilon ,$$

(16)

with D being the attractor dimension.

The predictive information [13] is the mutual information between the semi-infinite past and the future time series

$$PI\left(\epsilon \right)=\underset{m\to \infty }{lim}{PI}_m\left(\epsilon \right)$$

(17)

with

$$\begin{array}{cc}\hfill {PI}_m\left(\epsilon \right)& :=I({\overrightarrow{X}}_{t+m},\dots ,{\overrightarrow{X}}_{t+1};{\overrightarrow{X}}_t,\dots ,{\overrightarrow{X}}_{t-m+1};\epsilon )\hfill \end{array}$$

(18)

$$\begin{array}{cc}& =2H_m\left(\epsilon \right)-H_{2m}\left(\epsilon \right)\hfill \end{array}$$

If the limits Equations (14) and (17) exist the predictive information $PI\left(\epsilon \right)$ is equal to the excess entropy $E\left(\epsilon \right)$. For the finite time variants in general ${PI}_m\left(\epsilon \right)\ne E_m\left(\epsilon \right)$:

$${PI}_m=E_m+\sum _{k=m}^{2m-1}(h_{m-1}-h_k)\ge E_m.$$

However, if the system is Markov of order p the conditional probabilities will only depend on the previous p time steps, $p\left(\overrightarrow{X_t}\right|X_{t-1},\dots ,X_{t-\infty })=p\left({\overrightarrow{X}}_t\right|X_{t-1},\dots ,X_{t-p})$, hence $h_m=h_{m-1}$ for $m>p$ and therefore $E=E_{p+1}={PI}_{p+1}$.

2.2. Decomposing the Excess Entropy for Continuous States

In the literature [13,15,16] both the excess entropy and the predictive information were studied only for a given partition—usually a generating partition. Thus, using the excess entropy as a complexity measure for continuous valued time series has to deal with the fact that its value will be different for different partitions—even for different generating ones.

In Equation (16) we have seen that the excess entropy for deterministic systems becomes infinite in the limit $\epsilon \to 0$. The same applies to the predictive information defined in Equation (18). Moreover, we have seen that the increase of these complexity measures with decreasing ε is controlled by the attractor dimension of the system. Does this means that in the case of deterministic systems the excess entropy as a complexity measure reduces to the attractor dimension? Not completely. The constant in Equation (16) reflects not only the scale of the signal, but also statistical dependencies or memory effects in the signal, in the sense that it will be larger if the conditional entropies converge slower towards the KS-entropy.

How can we separate the different contributions? We will start by rewriting Equation (15) as a sum. Using the conditional entropies (Equation (13)) we get

$$E_m\left(\epsilon \right)=\sum _{k=0}^{m-1}(h_k\left(\epsilon \right)-h_{m-1}\left(\epsilon \right)).$$

Using the differences between the conditional entropies

$$\delta h_m\left(\epsilon \right):=h_{m-1}\left(\epsilon \right)-h_m\left(\epsilon \right)$$

(19)

the excess entropy can be rewritten as

$$E_m\left(\epsilon \right)=\sum _{k=1}^{m-1}k\delta h_k\left(\epsilon \right).$$

(20)

Note that the difference $\delta h_m\left(\epsilon \right)$ is the conditional mutual information

$$\delta h_m\left(\epsilon \right)=I({\overrightarrow{X}}_t:{\overrightarrow{X}}_{t-m}|{\overrightarrow{X}}_{t-1},\dots ,{\overrightarrow{X}}_{t-m+1}).$$

It measures dependencies over m time steps that are not captured by dependencies over $m-1$ time steps. In other words, how much uncertainty about $X_t$ can by reduced if in addition to the $m-1$ step past also the m th is taken into account. For a Markov process of order m-the $\delta h_k$ vanish for $k>m$. In this case the sum in Equation (20) contains only a finite number of terms. On the other side truncating the sum at finite m could be interpreted as approximating the excess entropy by the excess entropy of an approximating Markov process. What can be said about the scale dependency of the $\delta h_m\left(\epsilon \right)$? From Equations (2) and (3) follows that $h_m\left(\epsilon \right)\propto -log\left(\epsilon \right)$ for $m\le D$ and $h_m\left(\epsilon \right)\propto const$ for $m\ge D$. Using this and Equation (19) we have to distinguish four cases for deterministic systems. Note that $\delta D=D-\lfloor D\rfloor$ denotes the fractal part of the attractor dimension.

$$\delta h_m\left(\epsilon \right)\propto \left\{\begin{array}{ccc}\mathrm{const}& \text{if}& m\le D-1\hfill \\ -(1-\delta D)log\epsilon & \text{if}& m=\lfloor D\rfloor \hfill \\ -\delta Dlog\epsilon & \text{if}& m=\lfloor D\rfloor +1\hfill \\ \mathrm{const}& \text{if}& m>D+1.\hfill \end{array}\right.$$

(21)

Thus the sum Equation (20) can be decomposed into three parts:

$$E_m\left(\epsilon \right)=\sum _{k=1}^{\lfloor D\rfloor -1}k\delta h_k\left(\epsilon \right)+\sum _{k=\lfloor D\rfloor }^{\lfloor D\rfloor +1}k\delta h_k\left(\epsilon \right)+\sum _{\lfloor D\rfloor +2}^{m-1}k\delta h_k\left(\epsilon \right).$$

(22)

with the ε dependence showing up only in the middle term (MT):

$$\mathrm{MT}=\sum _{k=\lfloor D\rfloor }^{\lfloor D\rfloor +1}k\delta h_k\left(\epsilon \right)\propto c-log\frac{\epsilon }{{\epsilon }_D}$$

(23)

with ${\epsilon }_D$ denoting the length scale where the deterministic scaling range starts. Therefore we have decomposed the excess entropy in three terms: Two ideally ε independent terms and one ε-dependent term. The first term

$$E_{\mathrm{state}}=\sum _{k=1}^{\lfloor D\rfloor -1}k\delta h_k+c$$

(24)

will be called “state complexity” in the following because it is related to the information encoded in the state of the system. The constant c was added here in order to ensure that the ϵ-dependent term vanishes at ${ϵ}_D$—the beginning of the deterministic scaling range. The second ε-independent term

$$E_{\mathrm{mem}}=\sum _{\lfloor D\rfloor +2}^{m-1}k\delta h_k\left(\epsilon \right)$$

(25)

will be called “memory complexity” because it is related to the dependencies between the states on different time steps. What we call “state” in this context is related to the minimal embedding dimension to see the deterministic character of the dynamics which is $\lfloor D\rfloor +1$ [29]. In order to be able to get a one to one reconstruction of the attractor a higher embedding dimension might be necessary [32]. Both ε-independent terms together we will call “core complexity”

$$E_{\mathrm{core}}=E_{\mathrm{state}}+E_{\mathrm{mem}}.$$

(26)

So far we only addressed the case of a deterministic scaling range. In the case of a noisy chaotic system we have to distinguish two ε regions: the deterministic scaling range described above and the noisy scaling range with $h_m\left(\epsilon \right)\approx h_m^c-dlog\epsilon$ with $h_m^c$ determined by the noise level. In the stochastic scaling range all $\delta h_m$ become ε-independent and the decomposition Equation (22) seems to become unnecessary. This is not completely the case. Let us assume that the crossover between the two regions happens at a length scale ${\epsilon }^{*}$. Moreover, let us assume that for sufficiently large m we have in the deterministic scaling range $h_m\left(\epsilon \right)\approx h_{KS}$ (see Section 2.5 for an example). Then we have

$$h_m\left({\epsilon }^{*}\right)\approx h_{KS}\approx h_m^c-log{\epsilon }^{*}$$

(27)

which allows to express the cross-over scale ${\epsilon }^{*}$ in terms of the KS-entropy and the noise level related continuous entropy $h_m^c$

$$log{\epsilon }^{*}\approx h_m^c-h_{KS}.$$

(28)

Moreover, the excess entropy in the deterministic scaling range will behave as

$$E_m\left(\epsilon \right)\approx E_{state}+E_{mem}-Dlog\frac{\epsilon }{{\epsilon }_D}\text{for}\epsilon \ge {\epsilon }^{*}.$$

(29)

Evaluating this expression at the crossover length scale ${\epsilon }^{*}$ allows to express the value of the excess entropy in the stochastic scaling range as

$$E_m\approx E_{state}+E_{mem}+Dlog{\epsilon }_D-D(h_m^c-h_{KS})\text{for}\epsilon \le {\epsilon }^{*}.$$

(30)

In particular, this expression shows that decreasing the noise level, which increases $h_m^c$, will increase the asymptotic value of the excess entropy for noisy systems. Thus, an increased excess entropy or predictive information for a fixed length scale or partition can be achieved in many ways:

• by increasing dimension D of the dynamics
• by decreasing the noise level ${\epsilon }^{*}$
• by increasing the amplitude ${\epsilon }_D$
• by increasing the state complexity
• by increasing the correlations measured by the “memory” complexity, i.e., by increasing the predictability
• by decreasing the entropy rate $h_{KS}$, i.e., by decreasing the unpredictability

Naturally, the effect of the noise level will be observed in the stochastic scaling range only. In practice there might be more than one deterministic and stochastic scaling range or even no clear scaling range at all. How we will deal with these cases will be described below when we introduce our algorithm.

2.3. Methods for Estimating the Information Theoretic Measures

Reliably estimating entropies and mutual information is very difficult in high-dimensional spaces due to the increasing bias of entropy estimates. Therefore we will employ two different approaches. On the one hand we will use an algorithm for the estimation of the mutual information proposed by Kraskov et al. [26] based on nearest neighbor statistics which allows to reduce the bias by employing partitions of different sizes in spaces of different dimensions. On the other hand we calculate a proxy for the excess entropy using correlation entropies [25] of order $q=2$. These are related to the Rényi entropies of second order and the correlation sum provides an unbiased estimator. Both methods do not require binning but differ substantially in what they compute.

Estimation via local densities from nearest neighbor statistics (KSG) The most common approach to estimate information quantities of continuous processes, such as the mutual information, is to calculate the differential entropies (1) directly from the nearest neighbor statistics. The key idea is to use nearest neighbor distances [33,34,35] as proxies for the local probability density. This method corresponds in a way to an adaptive bin-size for each data point. For the mutual information $I(X,Y)$ (required e.g., to calculate the PI (Equation (18))), however, it is not recommended to naively calculate it directly from the individual entropies of X, Y and their joint $(X,Y)$ because they may have very dissimilar scale such that the adaptive binning leads to spurious results. For that reason a new methods was proposed by Kraskov et al. [26], that we call KSG, which only uses the nearest neighbor statistics in the joint space. We denote $I_{\mathrm{KSG}}^{\left(k\right)}(X,Y)$ the mutual information estimate where k nearest neighbors where used for the local estimation.

The length scale on which the mutual information is estimated by this algorithm depends on the available data. In the limit of infinite amount of data $I_{\mathrm{KSG}}^{\left(k\right)}(X,Y)={lim}_{\epsilon \to 0}I(X,Y,\epsilon )$ for $k\gg 1$. However, in order to evaluate the quantity at a certain length scale (similar to ε above) and assuming the same underlying space for X and Y, noise of strength η is added to the data resulting in

$$\begin{array}{c}\hfill I_{\mathrm{KSG}}^{\left(k\right)}(X,Y,\eta )=I_{\mathrm{KSG}}^{\left(k\right)}(X+U\left(\eta \right),Y+U\left(\eta \right))\end{array}$$

(31)

where $U\left(\eta \right)$ is the uniform distribution in the interval $[0,\eta ]$. The idea of adding noise is to make the processes X and Y independent within neighborhood sizes below the length scale of the noise. In this way only the structures above the added noise-level contribute to the mutual information. Note that for small η the actual scale (k-neighborhood size) may be larger due to sparsity of the available data.

Estimation via correlation sum The correlation sum is one of the standard tools in nonlinear time series analysis [36,37]. Normally it is used to estimate the attractor dimension. However, it can also be used to provide approximate estimates of entropies and derived quantities such as the excess entropy. The correlation entropies for a random variable $\overrightarrow{X}$ with measure $\mu \left(\overrightarrow{x}\right)$ are defined as [25]

$$H^{\left(q\right)}(\overrightarrow{X};\epsilon )=\left\{\begin{array}{cc}-\frac{1}{q-1}log\int \mu {\left(B(\overrightarrow{x},\epsilon )\right)}^{q-1}d\mu \left(\overrightarrow{x}\right)& \text{for}q\ne 1\\ -\int log\mu \left(B(\overrightarrow{x},\epsilon )\right)d\mu \left(\overrightarrow{x}\right)& \text{for}q=1\end{array}\right.$$

(32)

where $B(\overrightarrow{x},\epsilon )=\{\overrightarrow{y}\left|\right||\overrightarrow{y}-\overrightarrow{x}\left|\right|<\epsilon \}$ is the “ball” at $\overrightarrow{x}$ with radius ε. For $q=2$ Equation (32) becomes $H^{\left(2\right)}(\overrightarrow{X};\epsilon )=-log\int \mu \left(B(\overrightarrow{x},\epsilon )\right)d\mu \left(\overrightarrow{x}\right)$. The integral in this formula is also known as “correlation integral”. For N data points ${\overrightarrow{x}}_i\in {\mathbb{R}}^{n}i=1,\dots ,N$ it can be estimated using the correlation sum, which is the averaged relative number of pairs in an ε-neighborhood [36,37],

$$C^{\left(2\right)}\left(\epsilon \right)=\frac{2}{N(N-1)}\sum _{i=1}^{N-1}\sum _{j=i+1}^N\Theta \left(\epsilon -\parallel {\overrightarrow{x}}_i-{\overrightarrow{x}}_j\parallel \right).$$

(33)

Θ denotes the Heaviside function

$$\Theta \left(x\right)=\left\{\begin{array}{ccc}0& :& x<1\\ 1& :& x\ge 1.\end{array}\right.$$

Then the correlation entropy is

$$\begin{array}{cc}\hfill H^{\left(2\right)}\left(\epsilon \right)& =-log{C}^{\left(2\right)}\left(\epsilon \right).\hfill \end{array}$$

For sufficiently small ε it behaves as

$$\begin{array}{cc}\hfill H^{\left(2\right)}\left(\epsilon \right)& \approx \mathrm{const}-D^{\left(2\right)}log\epsilon \hfill \end{array}$$

with $D^{\left(2\right)}$ being the correlation dimension of the system [38]. A scale dependent correlation dimension can be defined as difference quotient

$$D^{\left(2\right)}\left(\epsilon \right)=log\frac{C^{\left(2\right)}(\epsilon +\Delta )}{C^{\left(2\right)}\left(\epsilon \right)}{\left(log\frac{\epsilon +\Delta }{\epsilon }\right)}^{-1}.$$

(34)

For a temporal sequence of states (or state vectors) $x_{i}i=1,\dots ,m$ we can now define block entropies $H_m^{\left(2\right)}\left(\epsilon \right)=-log{C}^{\left(2\right)}\left(\epsilon \right)$ by using $m\times d$-dimensional delay vectors ${\overrightarrow{x}}_i$. Now, we can define also the quantities corresponding to conditional entropies and to the excess entropy using the correlation entropy

$$\begin{array}{cc}\hfill h_m^{\left(2\right)}\left(\epsilon \right)& =H_{m+1}^{\left(2\right)}\left(\epsilon \right)-H_m^{\left(2\right)}\left(\epsilon \right)\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill E_m^{\left(2\right)}\left(\epsilon \right)& =H_m^{\left(2\right)}\left(\epsilon \right)-m{h}_{m-1}^{\left(2\right)}\left(\epsilon \right).\hfill \end{array}$$

(36)

We expect the same qualitative behavior regarding the ε-dependence of these quantities as for those based on Shannon entropies, see Equations (11) and (15). Quantitatively there might be differences, in particular for large ε and strongly non-uniform measures.

A comparison of the two methods with analytical results are given in the Appendix A, where we find a good agreement. Although, the KSG method seems to underestimate the mutual information for larger embeddings (higher dimensional state space). The correlation integral method uses a unit ball of diameter $2\epsilon$ whereas the KSG method measures the size of the hypercube enclosing k neighbors where the data was subject to additive noise in the interval $[0,\eta ]$. Thus comparable results are obtained with $\eta \approx 2\epsilon$.

2.4. Algorithm for Excess Entropy Decomposition

We are now describing the algorithm used to compute the proposed decomposition of the excess entropy in Section 2.2. The algorithm is composed of several steps: preprocessing, determining the middle term (MT) (Equation (23)), determining the constant in MT, and the calculation of the decomposition and of quality measures.

Preprocessing: Ideally the $\delta h$ curves are composed of straight lines in a log-linear representation, i.e., of the form $o-slog\left(\epsilon \right)$. We will refer to s as the slope (it is actually the inverted slope). Thus we perform fitting, that attempts to find segments following this form, details are provided in the Appendix B. Then the data is substituted by the fits in the intervals where the fits are valid. As for very small scales the $\delta h$ become very noisy we extrapolate below the fit with the smallest scale. In addition we calculate the derivative $\widehat{s}(m,\epsilon )=\frac{d\delta h_m}{d\epsilon }$ in each point, either from the fits (s, where available) or from finite differences of the data (using 5 points averaging).

Determining MT: In theory only two $\delta h_m$ should have a non-zero slope at each scale ε, see Equation (21). However, in practice we often have more terms, such that we need to find for each ε the maximal range $(m_l,m_u)$, where $\forall i\in [m_l,m_u]:\widehat{s}(i,\epsilon )>s_{\min }$, i.e., the slope is larger than the threshold $s_{\min }$. However, this is only valid for deterministic scaling ranges. In stochastic ranges all $\delta h$ should have zero slope. We introduce a measure of stochasticity, defined as $\kappa (m,\epsilon )=1-{\sum }_{k=m_l}^{m_u}\widehat{s}(k,\epsilon )$ which is 0 for purely deterministic ranges and 1 for stochastic ones. The separation between state and memory complexity is then inherited from the next larger deterministic range. Thus if $\kappa (m,\epsilon )\ge {\kappa }_{\max }$ we use $(m_l,m_u)$ at ${\epsilon }^{*}$, where ${\epsilon }^{*}={arg\; min}_{e\in (\epsilon ,\infty )}\kappa (m,e)<{\kappa }_{\max }$. Note that the here algorithmically defined ${ϵ}^{*}$ is not necessarily equal to the ${ϵ}^{*}$ defined above Equation (28) for an ideal-typical noisy deterministic system.

Determining the constant in MT: In order to obtain the scale-invariant constant c of the MT, see Equation (23), we would have to define a certain length scale ${\epsilon }_D$. Since this cannot be done robustly in practice (in particular because it may not be the same ${\epsilon }_D$ for each m) we resort to a different approach. The constant parts of the $\delta h_{m_l\dots m_u}$ terms in the MC can be determined from plateaus on larger scales. Thus, we define $c_m^{\mathrm{MT}}\left(\epsilon \right)={min}_{e\in (\epsilon ,{\epsilon }^{*}]}\delta h_m\left(e\right)$, where ${\epsilon }^{*}$ is smallest scale $>\epsilon$ where we have a near-zero slope, i.e., ${\epsilon }^{*}={arg\; min}_{e\in (\epsilon ,\infty )}s(m,e)<s_{\min }$. In case there is no such ${\epsilon }^{*}$ then $c_m^{\mathrm{MT}}=0$.

Decomposition and quality measures: The decomposition of the excess entropy follows Equations (24) and (25) with $m_l$ and $m_u$ used for splitting the terms:

$$\begin{array}{cc}\hfill E_{\mathrm{state}}& =\sum _{k=1}^{m_l-1}k\delta h_k+\sum _{k=m_l}^{m_u}k{c}_k^{\mathrm{MT}}{E}_{\epsilon }=\sum _{k=m_l}^{m_u}k(\delta h_k-c_k^{\mathrm{MT}})E_{\mathrm{mem}}=\sum _{k=m_u+1}^{m}k\delta h_k\hfill \end{array}$$

(37)

In addition we can compute different quality measures to indicate the reliability of the results, see Table 1.

Table 1. Quality measures for decomposition algorithm. We use the Iverson bracket for Boolean expression: $〚\mathrm{True}〛:=1$ and $〚\mathrm{False}〛:=0$. They are all normalized to $[0,1]$ where typically 0 is the best score and 1 is the worst.

Quantity	Definition	Description
κ=stochastic(ε)	$1-{\sum }_{k=m_l}^{m_u}\widehat{s}(k,\epsilon )$	0: fully deterministic, 1: fully stochastic at ε
% negative(ε)	$1/m{\sum }_{k=1}^m〚\delta h(k,\epsilon )<0〛$	percentage of negative $\delta h$
% no fits(ε)	$1/m{\sum }_{k=1}^m〚\mathrm{valid}\mathrm{fit}\mathrm{for}\delta h(k,\epsilon )〛$	percentage of $\delta h$ where no fit is available
% extrap.(ε)	$1/m{\sum }_{k=1}^m〚\mathrm{is}\delta h(k,\epsilon )\mathrm{extrapolated}〛$	percentage of $\delta h$ that where extrapolated

Table 1. Quality measures for decomposition algorithm. We use the Iverson bracket for Boolean expression: $〚\mathrm{True}〛:=1$ and $〚\mathrm{False}〛:=0$. They are all normalized to $[0,1]$ where typically 0 is the best score and 1 is the worst.

Quantity	Definition	Description
κ=stochastic(ε)	$1-{\sum }_{k=m_l}^{m_u}\widehat{s}(k,\epsilon )$	0: fully deterministic, 1: fully stochastic at ε
% negative(ε)	$1/m{\sum }_{k=1}^m〚\delta h(k,\epsilon )<0〛$	percentage of negative $\delta h$
% no fits(ε)	$1/m{\sum }_{k=1}^m〚\mathrm{valid}\mathrm{fit}\mathrm{for}\delta h(k,\epsilon )〛$	percentage of $\delta h$ where no fit is available
% extrap.(ε)	$1/m{\sum }_{k=1}^m〚\mathrm{is}\delta h(k,\epsilon )\mathrm{extrapolated}〛$	percentage of $\delta h$ that where extrapolated

2.5. Illustrative Example

To get an understanding of the quantities, let us first apply the methods to the Lorenz attractor, as a deterministic chaotic system, and to its noisy version as a stochastic system.

Deterministic system: Lorenz attractor The Lorenz attractor is obtained as the solution to the following differential equations:

$$\begin{array}{cc}\hfill \dot{x}& =s(-x+y)\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill \dot{y}& =-xz+rx-y\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill \dot{z}& =-xy-bz\hfill \end{array}$$

(40)

integrated with standard parameters $s=10$, $r=28$, $b=8/3$. Figure 1 displays the trajectory in phase-space ($x,y,z$) and using delay embedding of $x:\left(x\right(t),x(t-\tau \Delta ),\cdots ,x(t-m\tau \Delta \left)\right)$ using $m=3$ and time delay is $\tau =10$ (for sampling time $\Delta =0.01$). The equations where integrated with Runge-Kutta fourth order with step size $5\times {10}^{-4}$. An alternative time delay may be obtained from first minimum of mutual information as a standard choice [39,40], which would be $\tau =18$, but $\tau =10$ yields clearer results for presentation.

Figure 1. Lorenz attractor with standard parameters. (a) Trajectory in state space $(x,y,z)$. (b) Embedding of x with $m=3$, time-delay $\tau =10$, and $\Delta =0.01$.

Figure 1. Lorenz attractor with standard parameters. (a) Trajectory in state space $(x,y,z)$. (b) Embedding of x with $m=3$, time-delay $\tau =10$, and $\Delta =0.01$.

The result of the correlation sum method using TISEAN [41] is depicted in Figure 2 for ${10}^6$ data points. From the literature, we know that the chaotic attractor has a fractal dimension of $2.06$ [38] which we can verify with the correlation dimension $D_m^{\left(2\right)}$, see Figure 2a for $m>2$ and small ε. The conditional $h_m^{\left(2\right)}$ (d) becomes constant and identical for larger m. The excess entropy $E_m^{\left(2\right)}$ (g) approaches the predicted scaling behavior of: $\mathrm{const}-Dlog\left(\epsilon \right)$ (Equation (16)) with $D=2.06$ and $\mathrm{const}=4.75$. The predictive information ${PI}_m$. (j) shows the same scaling behavior on the coarse range but with a smaller slope and saturates at small ε.

Stochastic system: noisy Lorenz attractor In order to illustrate the fundamental differences between deterministic and stochastic systems when analyzed with information theoretic quantities we consider now the Lorenz dynamical system with dynamic noise (additive noise to the state ($x,y,z$) in Equations (38)–(40) before each integration step) as provided by the TISEAN package [41]. The dimension of a stochastic systems is infinite, i.e., for embedding m the correlation integral yields the full embedding dimension as shown in Figure 2b,c for small ε.

On large scales the dimensionality of the underlying deterministic system can be revealed to some precision either from $D^{\left(2\right)}$ or from the slope of $E^{\left(2\right)}$ or PI. Thus, we can identify a determinstic and a stochastic scaling range with different qualitative scaling behavior of the quantities. In contrast with the deterministic case, the excess entropy converges in the stochastic scaling range and no more contribution from $m>5$ are observed as shown in Figure 2h,i. By measuring the ε-dependence (slope) we can actually differentiate between deterministic and stochastic systems and scaling ranges, which is performed by the algorithm Section 2.4 and presented in Figure 3. The predictive information Figure 2k,l again yields a lower slope, meaning a lower dimension estimate, but is otherwise consistent. However, it does not allow to safely distinguish determinstic and stochastic systems, because it always saturates due to the effect of finite amount of data.

Figure 2. Correlation dimension $D_m^{\left(2\right)}$ (Equation (34)) (a)–(c), conditional block entropies $h_m^{\left(2\right)}$ (Equation (35)) (d)–(f), excess entropy $E_m^{\left(2\right)}$ (Equation (36) (g)–(i), and predictive information ${PI}_m$ (Equation (18)) (j)–(l) of the Lorenz attractor estimated with the correlation sum method (a)–(i) and KSG (j)–(l) for the deterministic system (first column) and with absolute dynamic noise $0.005$, and $0.01$ (second and third column respectively). The error bars in (j)–(l) show the values calculated on half of the data. All quantities are given in nats (natural unit of information with base e) and in dependence of the scale (ε,η in space of x (Equation (38))) for a range of embeddings m, see color code. In (d)–(f) the fits for $h_0=H_1$ allow to determine $H^C=2.65$ and for $h_m$ give $h_{KS}$ and $h_m^c$, Equations (27)–(30). Parameters: delay embedding of x with $\tau =10$.

Figure 2. Correlation dimension $D_m^{\left(2\right)}$ (Equation (34)) (a)–(c), conditional block entropies $h_m^{\left(2\right)}$ (Equation (35)) (d)–(f), excess entropy $E_m^{\left(2\right)}$ (Equation (36) (g)–(i), and predictive information ${PI}_m$ (Equation (18)) (j)–(l) of the Lorenz attractor estimated with the correlation sum method (a)–(i) and KSG (j)–(l) for the deterministic system (first column) and with absolute dynamic noise $0.005$, and $0.01$ (second and third column respectively). The error bars in (j)–(l) show the values calculated on half of the data. All quantities are given in nats (natural unit of information with base e) and in dependence of the scale (ε,η in space of x (Equation (38))) for a range of embeddings m, see color code. In (d)–(f) the fits for $h_0=H_1$ allow to determine $H^C=2.65$ and for $h_m$ give $h_{KS}$ and $h_m^c$, Equations (27)–(30). Parameters: delay embedding of x with $\tau =10$.

Figure 3. Excess Entropy decomposition for the Lorenz attractor into state complexity (blue shading), memory complexity (red shading), and ε-dependent part $E_{\epsilon }$ (beige shading) (all in nats). Columns as in Figure 2: deterministic system (a), with absolute dynamic noise $0.005$ (b), and $0.01$ (c). A set of quality measures and additional information is displayed on the top using the right axis. $m_l$ and $m_u$ refer to the terms in $E_{\epsilon }$ (Equation (37)). Scaling ranges that are identified as stochastic are shaded in gray (stochastic indicator $>{\kappa }_{\max }=0.5$). In manually chosen ranges, marked with vertical black lines, we evaluate the mean and standard deviation of the memory- and state complexity. Parameters: $s_{\min }=0.1$.

Figure 3. Excess Entropy decomposition for the Lorenz attractor into state complexity (blue shading), memory complexity (red shading), and ε-dependent part $E_{\epsilon }$ (beige shading) (all in nats). Columns as in Figure 2: deterministic system (a), with absolute dynamic noise $0.005$ (b), and $0.01$ (c). A set of quality measures and additional information is displayed on the top using the right axis. $m_l$ and $m_u$ refer to the terms in $E_{\epsilon }$ (Equation (37)). Scaling ranges that are identified as stochastic are shaded in gray (stochastic indicator $>{\kappa }_{\max }=0.5$). In manually chosen ranges, marked with vertical black lines, we evaluate the mean and standard deviation of the memory- and state complexity. Parameters: $s_{\min }=0.1$.

Let us now consider the decomposition of the excess entropy as discussed in Section 2.2 and Section 2.4 to see whether the method is consistent. Figure 3 shows the scale-dependent decomposition together with the determined stochastic and deterministic scaling ranges. The resulting values for comparing the determinstic scaling range are given in Table 2 and reveal, that the $E_{\mathrm{core}}$ decreases for increasing noise level similarly as the constant reduces. This is a consistency check because the intrinsic scale (amplitudes) of the deterministic and noisy Lorenz system are almost identical. We also checked the decomposition for a scaled version of the data where we found the same values state-, memory- and core complexity. We see also that the state complexity $E_{\mathrm{state}}$ vanishes for larger noise, because the dimension drops significantly below 2, where we have no summands in the state complexity and the constant $c^{\mathrm{MT}}$ is also 0.

Table 2. Decomposition of the excess entropy for the Lorenz system. Presented are the unrefined constant and dimension of the excess entropy, the state-, memory-, and core complexity (in nats) determined in the deterministic scaling range, see Figure 3.

		Determ.	Noise $0.005$	Noise $0.01$	Noise $0.02$
Equation (16)	D	2.11	1.95	1.68	1.59
Equation (16)	$\mathrm{const}$	4.62	4.55	4.37	4.21
Equation (27)	${\epsilon }^{*}$	0	0.12	0.23	0.45
Equation (24)	$E_{\mathrm{state}}$	0.68 ± 0	0.68 ± 0	0	0
Equation (25)	$E_{\mathrm{mem}}$	1.86 ± 0.06	1.27 ± 0.02	1.98 ± 0.11	1.2 ± 0.1
Equation (26)	$E_{\mathrm{core}}$	2.54 ± 0.06	1.95 ± 0.02	1.98 ± 0.11	1.2 ± 0.1

Table 2. Decomposition of the excess entropy for the Lorenz system. Presented are the unrefined constant and dimension of the excess entropy, the state-, memory-, and core complexity (in nats) determined in the deterministic scaling range, see Figure 3.

		Determ.	Noise $0.005$	Noise $0.01$	Noise $0.02$
Equation (16)	D	2.11	1.95	1.68	1.59
Equation (16)	$\mathrm{const}$	4.62	4.55	4.37	4.21
Equation (27)	${\epsilon }^{*}$	0	0.12	0.23	0.45
Equation (24)	$E_{\mathrm{state}}$	0.68 ± 0	0.68 ± 0	0	0
Equation (25)	$E_{\mathrm{mem}}$	1.86 ± 0.06	1.27 ± 0.02	1.98 ± 0.11	1.2 ± 0.1
Equation (26)	$E_{\mathrm{core}}$	2.54 ± 0.06	1.95 ± 0.02	1.98 ± 0.11	1.2 ± 0.1

To summarize, in deterministic systems with a stochastic component, we can identify different ε-ranges (scaling ranges) within which stereotypical behaviors of the entropy quantities are found. This allows us to determine the dimensionality of the deterministic attractor and decompose the excess entropy in order to obtain two ε-independent complexity measures.

3. Results

In this section, we will apply our methods to the behavior of robotic systems. We use a previously published control scheme that can generate a large variety of smooth and coherent behaviors from a local learning rule [11,42]. As it is a simplified version of the time-local predictive information maximization controller we aim at answering the hypothesis that the controller actually increases the complexity of behavior during learning.

3.1. Application to Robotics: Controller and Learning

Let us now briefly introduce the control framework and the robots used to generate the behavior we are analyzing in the following. The robots are simulated in a physically realistic rigid-body simulation [43]. Each robot has a number of sensors producing real-valued stream $s\in {\mathbb{R}}^n$ and a number of actuators, also controlled by real values $a\in {\mathbb{R}}^o$. The sensors measure the actual joint angles and the body accelerations. In all examples the actuators are position controlled servo motors with the particular property to have no power around the set-point. This enables small perturbations to be perceivable in the joint-position sensor values. The time is discretized with an update frequency of 50 Hz.

The control is done in a reactive closed-loop fashion by a one layer feed-forward neural network:

$$\begin{array}{c}\hfill a_t=K(s_t,C,h)=tanh(C{s}_t+h),\end{array}$$

(41)

where C is a weight matrix, h is a weight vector, and tanh is applied component-wise. For fixed weights this controller is very simple, however it can generate non-trivial behavioral modes due to the complexity of the body-environment interaction. For instance with a scaled rotation matrix C an oscillatory behavior may be controlled which frequency is adaptive to how fast the physical system follows. To get a variety of behavioral modes the parameters ($C,h$) have to be changed which can be done by optimizing certain objective functions. Several methods have been proposed that use generic (robot independent) internal drives. Homeokinesis [8,44] for instance balances dynamical instability with predictability. More recently the maximization of the time-local predictive information of the sensor process was used as driving force [9]. On an intuitive level, the maximization of the PI (Equation (18)) of the sensor process leads to a high variety of sensory input due to the entropy term $H\left(S\right)$ while keeping a temporal structure by minimizing the future uncertainty reflected in the conditional entropy $H\left(S^{\prime }\right|S)$. Indeed we have found that it produces smooth and coordinated behavior in different robotic systems, including hexapod robots and humanoids. On the other hand, maximizing PI also leads to behavior of high complexity with maximal dimension as discussed in Section 2.2. A simplified version of the PI-maximizing algorithm, called ULR published in [11,42], leads to more coherent and defined behavioral modes of seemingly low dimensionality which we use in this paper. In order to apply the method to behavior of robots or possibly to that of animals the state space reconstruction has to be done from certain measurements. Here we use a single sensor value, but the formalism can be extended to multiple sensor values as well. It must be guaranteed that sufficient physical coupling exist between the measured quantity and the rest of the body.

3.2. Experiments

Let us now present the emergent behaviors obtained from the control algorithm ULR. We use two robots, a snake-like robot and an insect-like hexapod robot as illustrated in Figure 4. The behaviors of the snake are published here for the first time, whereas the behaviors of the hexapod have been published before in Der and Martius [11]. For an overview and links to the videos see Table 3. However, this paper focuses primarily on the quantification of behavior and not on its generation. The Snake has identical cylindrical segments which are connected by actuated 2 degrees of freedom (DoF) universal joints. In this way two neighboring segments can be inclined to each other, but without relative torsion along the segment-axes. The sensor values are the actual positions of the joints. If the robot is controlled with the ULR controller very quickly highly coordinated behavioral modes emerge, such as crawling or side-rolling, see Figure 5. An external perturbation can bring the robot into another behavior. Without perturbation the robot typically transitions into one behavior and remain in it. We recorded $9\times {10}^5$ samples from a typically side-rolling and crawling behavior.

Table 3. Experiments, data sets and videos, see [18].

Robot	Behavior	Length	Dataset	Video
Snake	side rolling	$9\times {10}^5$	D-S1	Video S1
Snake	crawling	$9\times {10}^5$	D-S2	Video S2
Hexapod	jumping after 2 min	${10}^6$	D-H1	Video H1
Hexapod	jumping after 4 min	${10}^6$	D-H2	Video H2
Hexapod	jumping after 8 min	${10}^6$	D-H3	Video H3

Table 3. Experiments, data sets and videos, see [18].

Robot	Behavior	Length	Dataset	Video
Snake	side rolling	$9\times {10}^5$	D-S1	Video S1
Snake	crawling	$9\times {10}^5$	D-S2	Video S2
Hexapod	jumping after 2 min	${10}^6$	D-H1	Video H1
Hexapod	jumping after 4 min	${10}^6$	D-H2	Video H2
Hexapod	jumping after 8 min	${10}^6$	D-H3	Video H3

Figure 4. Robots used for the experiments. (a) The Hexapod . 18 actuated DoF: 2 shoulder and 1 knee joint per leg. (b) The Snake . 14 DoF: 8 segments connected by 2 DoF joints each.

Figure 4. Robots used for the experiments. (a) The Hexapod . 18 actuated DoF: 2 shoulder and 1 knee joint per leg. (b) The Snake . 14 DoF: 8 segments connected by 2 DoF joints each.

Figure 5. Side-rolling and crawling of the Snake . Note that also for rolling each actuator has to act accordingly because no torsion is possible, see Figure 4b.

Figure 5. Side-rolling and crawling of the Snake . Note that also for rolling each actuator has to act accordingly because no torsion is possible, see Figure 4b.

The Hexapod is a insect-like robot with 18 DoF, see Figure 4a. The sensor values are the joint angles and the vertical velocity of the trunk. Note, that the actual phase space dimension is $18+18+4=40$ as we have joint angles, joint velocities, and height and orientation of the trunk. Controlled with the ULR controller the robot develops a jumping behavior, see Figure 6, within 8 min of simulated time [11]. The behavior of the robot changes due to the update dynamics of the controller. For the analysis a stationary process is required, such that we stop the update at some time and obtain a static controller (fixed $C,h$). Using this the robots behavior stays essentially the same. We analyze three behaviors, where the learning was stopped at 2, 4 and 8 min after the start and recorded each ${10}^6$ samples. After “2 min” the robot moves periodically up and down. The “4 min” behavior shows already signs of jumping and the final behavior after “8 min” is a whole body periodic jumping as displayed in Figure 6.

Figure 6. Jumping motion pattern emerging with vertical speed sensor 8 min after the start.

Figure 6. Jumping motion pattern emerging with vertical speed sensor 8 min after the start.

3.3. Quantifying the Behavior

We start with analyzing the data from the Snake experiments followed by the Hexapod experiments.

3.3.1. Snake

In Figure 7 the result of the quantification of the two behaviors (Figure 5) are presented. We used the delay embedding of the sensor $x_7$ which is one of the joint angular sensors of the middle joint. We find that the side-rolling behavior is roughly 1-dimensional on the behaviorally relevant scales ($0.05<\epsilon <0.5$), see Figure 7c. On smaller scales the dimension raises with each embedding although the embedding dimension is not reached on the observable length scales and therefore the excess entropy does not saturate. On the larger scales we find the expected scaling behavior given by Equation (16) with $D\approx 1$.

The crawling behavior is also 1-dimensional on the coarse length scales ($0.1<\epsilon <0.5$), see Figure 7d. The smaller scales, however, bear a surprise because the dimension raises to a value of $\approx 2.5$. The excess entropy shows also a slope of $D\approx 1.0$ on the coarse length scales

Calculating the predictive information using the KSG method yields a similar result on the large scales, as shown in Figure 7g,h. For smaller scales, however, the predictive information always saturates, as seen before in Figure 2j. In the crawling case we get again an underestimated dimension of $D\approx 2$.

Figure 7. Quantification of two Snake behaviors. First column: side rolling; second column: crawling (see Figure 5(a,b). Phase plots with Poincaré sections at the maximum of first and second embedding. (e,f) Excess entropy with decomposition (in nats), see Figure 3 for more details. (g,h) Predictive information (in nats), see Figure 2 for more details. Parameters: delay embedding of sensor $x_7$ (middle segment) with $\tau =8$, $s_{\min }=0.05$, $\kappa =0.5$.

Figure 7. Quantification of two Snake behaviors. First column: side rolling; second column: crawling (see Figure 5(a,b). Phase plots with Poincaré sections at the maximum of first and second embedding. (e,f) Excess entropy with decomposition (in nats), see Figure 3 for more details. (g,h) Predictive information (in nats), see Figure 2 for more details. Parameters: delay embedding of sensor $x_7$ (middle segment) with $\tau =8$, $s_{\min }=0.05$, $\kappa =0.5$.

The decomposition of the excess entropy follows for the above experiments are given in Table 4. Interestingly, the $E_{\mathrm{core}}$ attributes a significantly higher complexity to the crawling behavior, whereas the plain excess entropy constant would suggest that side-rolling is more complex. Since we take out the effect of different scales our new quantities are more trustworthy.

Table 4. Decomposition of the excess entropy for the Snake. Dimension and unrefined constant, state-, memory-, and core complexity (in nats) on the coarse deterministic scaling range $\epsilon \in (0.05,0.6)$.

		Side Rolling	Crawling
Equation Equation (16)	D	1.07	1.15
Equation Equation (16)	$\mathrm{const}$	0.7	0.53
Equation Equation (24)	$E_{\mathrm{state}}$	0.0 ± 0	0 ± 0
Equation Equation (25)	$E_{\mathrm{mem}}$	0.69 ± 0.14	0.94 ± 0.16
Equation Equation (26)	$E_{\mathrm{core}}$	0.69 ± 0.14	0.94 ± 0.16

Table 4. Decomposition of the excess entropy for the Snake. Dimension and unrefined constant, state-, memory-, and core complexity (in nats) on the coarse deterministic scaling range $\epsilon \in (0.05,0.6)$.

		Side Rolling	Crawling
Equation Equation (16)	D	1.07	1.15
Equation Equation (16)	$\mathrm{const}$	0.7	0.53
Equation Equation (24)	$E_{\mathrm{state}}$	0.0 ± 0	0 ± 0
Equation Equation (25)	$E_{\mathrm{mem}}$	0.69 ± 0.14	0.94 ± 0.16
Equation Equation (26)	$E_{\mathrm{core}}$	0.69 ± 0.14	0.94 ± 0.16

Note, that for the crawling behavior at the small scaling range ($\epsilon <0.01$) we can also perform the decomposition and obtain $E_{\mathrm{state}}=0.84\pm 0.17$, $E_{\mathrm{mem}}=3.23\pm 0.46$, however these values are mostly useful for comparison, which we only have on the large scale. Note that $E_{\mathrm{state}}$and $E_{\mathrm{mem}}$are not really constant in the plots because the deterministic scaling range is not very good in this case (cf. the quality measure “no fits” in Figure 7f).

Figure 8 shows the quantification of the three successive behaviors of the Hexapod robot, reconstructed from the delay embedding of sensor $x_1$ (up-down direction of right hind leg). We clearly see an increase in dimensionality of the behavior, Figure 8d–f which we can expect because the high jumping behavior requires more compensation movements due to different landing poses. For the first behavior we can identify three scaling ranges of different dimensionality and complexity. On the coarse level $[0.1,0.6]$ the behavior is described by a dimension of $1.2$ whereas on medium scales $[0.006,0.02]$ we find an approximately $1.6$ dimensional behavior and on the small scales $[0.001,0.005]$ it appears 5 dimensional.

For the second behavior we find only a short plateau in Figure 8e at $[0.06,0.3]$ of dimensionality 4. The excess entropy shows a lower dimensionality of $\approx 3.4$, see Figure 8h. For the third behavior we cannot identify a clear plateau in the dimension plot. The excess entropy allows us to attribute a dimensionality and core complexity to the behavior on large scales, as summarized in Table 5. Whereas the scaling behavior of the excess entropy suggests a similar dimensionality, the core complexity suggest that the 8 min behavior is more complex.

Figure 8. Quantification of the three Hexapod behaviors. Behaviors: first column: 2 min; second column: 4 min; third column 8 min, see Figure 6. (a)–(c) Phase plots with Poincaré sections at the maximum and minimum of first and second embedding. The color of the trajectory (blue–black) encodes for the third embedding dimension. (g)–(i) Excess entropy with decomposition (in nats), see Figure 3 for more details. The excess entropy fit at the smallest scale in (g) is $-18-5log\left(\epsilon \right)$. (j)–(l) Predictive information (in nats), see Figure 2 for more details. Parameters: delay embedding of sensor $x_1$ (up-down direction of right hind leg) with $\tau =4$, $s_{\min }=0.05$, ${\kappa }_{\max }=0.5$.

Figure 8. Quantification of the three Hexapod behaviors. Behaviors: first column: 2 min; second column: 4 min; third column 8 min, see Figure 6. (a)–(c) Phase plots with Poincaré sections at the maximum and minimum of first and second embedding. The color of the trajectory (blue–black) encodes for the third embedding dimension. (g)–(i) Excess entropy with decomposition (in nats), see Figure 3 for more details. The excess entropy fit at the smallest scale in (g) is $-18-5log\left(\epsilon \right)$. (j)–(l) Predictive information (in nats), see Figure 2 for more details. Parameters: delay embedding of sensor $x_1$ (up-down direction of right hind leg) with $\tau =4$, $s_{\min }=0.05$, ${\kappa }_{\max }=0.5$.

3.3.2. Hexapod

The predictive information calculated with the KSG method underestimates the dimension by far and is only able to report structure on the coarsest scale, see Figure 8j–l and compare for instance slope $1.8$ of PI in (k) with slope $3.36$ of $E^{\left(2\right)}$ in (h). Note that usually only the value of the mutual information without added noise (or only very small one) is reported (left-most value in our graphs). In our case this would attribute the highest complexity to the 2 min behavior and the lowest to the 8 min behavior, see also Table 5. This is because the intrinsic noise level increases from 2 min to 8 min, such that the plateau level is much lower.

Table 5. Decomposition of the excess entropy for the Hexapod behaviors on the fine and coarse scale. For the 4 min and 8 min behavior we have no reliable estimate on the fine scale. The core complexity and the dimension rises from 2 min to 4 min. From 4 min to 8 min the dimension stayed the same but the core complexity increases. The last line provides the predictive information on the smallest scale.

		2 min (Fine)	2 min (Coarse)	4 min (Coarse)	8 min (Coarse)
	ε-Range	$(\mathbf{0}.\mathbf{006},\mathbf{0}.\mathbf{025})$	$(\mathbf{0}.\mathbf{1},\mathbf{0}.\mathbf{4})$	$(\mathbf{0}.\mathbf{05},\mathbf{0}.\mathbf{2})$	$(\mathbf{0}.\mathbf{1},\mathbf{0}.\mathbf{4})$
Equation (16)	D	1.6	1.25	3.36	3.45
Equation (16)	$\mathrm{const}$	−0.09	−0.2	−0.81	0.03
Equation (24)	$E_{\mathrm{state}}$	0.7 ± 0.11	0 ± 0	0.17 ± 0.03	0 ± 0
Equation (25)	$E_{\mathrm{mem}}$	2.1 ± 0.27	0.85 ± 0.16	3.74 ± 0.54	4.79 ± 1.03
Equation (26)	$E_{\mathrm{core}}$	2.8 ± 0.38	0.85 ± 0.16	3.91 ± 0.57	4.79 ± 1.03
Equation (18)	$PI(\eta =0)$	$\approx 9$	-	$\approx 5.5$	$\approx 4$

Table 5. Decomposition of the excess entropy for the Hexapod behaviors on the fine and coarse scale. For the 4 min and 8 min behavior we have no reliable estimate on the fine scale. The core complexity and the dimension rises from 2 min to 4 min. From 4 min to 8 min the dimension stayed the same but the core complexity increases. The last line provides the predictive information on the smallest scale.

		2 min (Fine)	2 min (Coarse)	4 min (Coarse)	8 min (Coarse)
	ε-Range	$(\mathbf{0}.\mathbf{006},\mathbf{0}.\mathbf{025})$	$(\mathbf{0}.\mathbf{1},\mathbf{0}.\mathbf{4})$	$(\mathbf{0}.\mathbf{05},\mathbf{0}.\mathbf{2})$	$(\mathbf{0}.\mathbf{1},\mathbf{0}.\mathbf{4})$
Equation (16)	D	1.6	1.25	3.36	3.45
Equation (16)	$\mathrm{const}$	−0.09	−0.2	−0.81	0.03
Equation (24)	$E_{\mathrm{state}}$	0.7 ± 0.11	0 ± 0	0.17 ± 0.03	0 ± 0
Equation (25)	$E_{\mathrm{mem}}$	2.1 ± 0.27	0.85 ± 0.16	3.74 ± 0.54	4.79 ± 1.03
Equation (26)	$E_{\mathrm{core}}$	2.8 ± 0.38	0.85 ± 0.16	3.91 ± 0.57	4.79 ± 1.03
Equation (18)	$PI(\eta =0)$	$\approx 9$	-	$\approx 5.5$	$\approx 4$

4. Discussion

In recent years research in autonomous robots has been more and more successful in developing algorithms for generating behavior from generic task-independent objective functions [1,2,3,4,5,6,7,9]. This is valuable, on the one hand, as a step towards understanding and creating autonomy in artificial beings, and on the other hand, to obtain eventually more effective learning algorithms for mastering real-life problems. This is because the task-agnostic drives aim at efficient exploration of the agent’s possibilities by generically useful behavior [6,8,9,45,46] and at generating an operational state of the system [47] from which specific tasks can quickly be accomplished.

Very often emerging behaviors are not the result of optimizing a global function but for instance self-organize from local interaction rules. We propose to use information theoretic quantities for quantifying these behaviors. There are already studies using information theoretic measures to characterize emergent behavior of autonomous robots [48,49,50]—in particular with the aim to use them as task-independent objective functions [1,2,3,4,5,9]. Usually these information theoretic quantities are estimated for a fixed partition, i.e., a specific discretization, or by using differential entropies assuming that the data were sampled from a probability density. In this paper, we show that it is both necessary and fruitful to explicitly take into account the scale dependence of the information measures.

We quantify and compare behaviors emerging from self-organizing control of high-dimensional robots by estimating the excess entropy and the predictive information of their sensor time series. The estimation is done on the one hand using entropy estimates based on the correlation integral and on the other hand the predictive information is directly estimated as a mutual information using an estimator (KSG) based on differential entropies. The latter implicitly assumes stochastic data, i.e., the existence of a smooth probability density for the temporal joint distribution, such that the predictive information converges towards a finite value. However, this is not the case for deterministic systems. Due to the adaptivity of the algorithm the predictive information is then calculated at a certain length scale which depends on the available amount of data. Thus, a naive application of the KSG estimator can lead to totally meaningless results, because it does not allow to control for the length scale. To avoid this problem and to control the length scale, we applied additive noise. Nevertheless, our results show that the complexity estimates based on the KSG estimator are much less reliable than those based on the correlation integral. While for simple stochastic systems such as autoregressive models both estimators deliver consistent results for small embedding dimensions, the predictive information is underestimated due to finite sample effects for larger embedding dimensions (cf. Figure A1). For more complex systems such as the Lorenz attractor or the behavioral data from robots the KSG algorithm underestimates the dimensionality, see e.g., Figure 8j–l, does not allow to extract length scales properly, and blurs the distinction between stochastic and deterministic scaling ranges.

An important result of the presented study is that the complexity of the behavior can be different at different length scales: an example is the crawling behavior of the Snake which is 1-dimensional on the coarse scale and 2.6-dimensional on the small scales, see Figure 7 (the same for the Hexapod). In order to quantify these effects we analyzed for the first time the resolution dependence of the well-known complexity measure for time series known as excess entropy [14,15], effective measure complexity [16], or predictive information [13]. We show that the scale dependent excess entropy reflects different properties of the system: scale, dimension and correlations. For deterministic systems the attractor dimension controls how the complexity increases with increasing resolution, while the offset depends on the overall scale of the behavior, e.g., the amplitude of an oscillation, and on the structure of the temporal correlations. In order to disentangle the latter two effects we introduced a new decomposition of the excess entropy with the core complexity quantifying the structural complexity beyond the dimension. In order to estimate it one has to remove the effect of the overall scale of the behavior from the excess entropy. We devised an algorithm based on an automatic identification of scaling ranges in the conditional mutual information $\delta h_m\left(ϵ\right)$.

For noisy systems the excess entropy remains finite. The actual value, however, depends on the noise level, which is consistent with the intuition: The larger the noise the lower the complexity if all other factors remain constant. While we have already a good understanding of these quantities for systems with clearly identifiable scaling regions as in the noisy Lorenz system, very often real systems do not show such ideal types of behavior. Therefore we had to use heuristic procedures which provide plausible results for the data under study. Our quantification of the behaviors of the Snake found that crawling has a higher complexity than side-winding, even though the original excess entropy would have suggested the opposite. For the Hexapod as a learning system (aiming at maximizing time local PI) we were able to quantify the learning progress: The decomposition of the excess entropy revealed the strategy used to increase the complexity: (1) The dimensionality of the behavior increased on the large scales. This does not necessarily mean that the attractor dimension in the mathematical sense increased, because the latter is related to behavior on infinitely small scales which might be irrelevant for the character of the behavior. (2) The core complexity is increased on the large scales indicating also more complex temporal correlations in the learned behavior. We also discuss further strategies for increasing the complexity of a time-series, that could be employed in designing future learning algorithms, most notably, the decrease in the entropy rate and the increase of the state complexity.

Our study contributes a well grounded and practically applicable quantification measure for the behavior of autonomous systems as well as time-series in general. In addition, a set of quality indicators are provided for self-assessment in order to avoid spurious results.

Future work is to develop a better theoretical understanding of the scale dependence of the excess entropy for realistic behaviors that are not low dimensional deterministic or linearly stochastic such as the behavior observed in the Hexapod after 8 minutes of learning, Figure 8. We expect also further insights by applying the same kind of analysis to movement data from animals and humans.