Intrinsic Motivation Based Hierarchical Exploration for Model and Skill Learning

Abstract

Hierarchical skill learning is an important research direction in human intelligence. However, many real-world problems have sparse rewards and a long time horizon, which typically pose challenges in hierarchical skill learning and lead to the poor performance of naive exploration. In this work, we propose an algorithmic framework called surprise-based hierarchical exploration for model and skill learning (Surprise-HEL). The framework leverages the surprise-based intrinsic motivation for improving the efficiency of sampling and driving exploration. It also combines the surprise-based intrinsic motivation and the hierarchical exploration to speed up the model learning and skill learning. Moreover, the framework incorporates the reward independent incremental learning rules and the technique of alternating model learning and policy update to handle the changing intrinsic rewards and the changing models. These works enable the framework to implement the incremental and developmental learning of models and hierarchical skills. We tested Surprise-HEL on a common benchmark domain: Household Robot Pickup and Place. The evaluation results show that the Surprise-HEL framework can significantly improve the agent’s efficiency in model and skill learning in a typical complex domain.

1. Introduction

In the field of psychology and neuroscience, one of the core objectives is to understand the formal structure of behavior [1,2]. It has been shown that the behavior displays a hierarchical structure in humans and animals. Simple actions are combined into coherent subtasks, and these subtasks are further combined to achieve higher-level goals [3]. The hierarchical structures are observed readily in our daily life: opening a faucet is the first step of the task ‘cleaning vegetables’, which is part of the bigger task ‘cooking a dish’. The detailed formal analysis of the hierarchical structure of behavior can be found in Whiten’s paper [4].

Hierarchical structure-based learning and planning are one of the most important research fields in artificial intelligence. Compared with flat representation, hierarchical representation is more efficient and compact, which is computationally simpler for encoding the complex behaviors at the neural level [5]. Moreover, the hierarchical representation can accelerate the discovery of new hierarchical behaviors through planning and learning [6,7,8,9].

Exploration is the fundamental of hierarchical learning and planning. However, exploration in environments with sparse rewards is challenging, as exploration over the space of primitive action sequences is unlikely to result in a reward signal. For humans, the problem of sparse rewards is solved by intrinsic motivation based exploration [10,11,12]. Intrinsic motivation is defined as the doing of an activity for its own sake, rather than to directly solve some specific problems [10]. When people are intrinsically motivated, they act for fun or challenge, not for external products, stress or rewards [11]. Intrinsic motivation based exploration can explore the environment and discover hierarchical skills for its own sake. Moreover, the learned skills will be reused to solve more complex and unfamiliar problems in later life [12]. In artificial intelligence, such intrinsic motivation based exploration is efficient and scalable to facilitate skill learning, and it can eventually help the agent solve the problems posed by the environment in their lifelong time.

In this work, we adopted surprise as an effective and robust intrinsic motivator to drive exploration. Surprise has played crucial roles in human behavior. The theories of intrinsic motivation propose that surprise is one of the main factors that motivate exploration, arouse interest, and drive learning [13]. There is a comprehensive agreement that surprise is an emotion caused by a mismatch between actual observations or experiences and expectations [14]. In this paper, we employed the dictionary definition of “surprise” as “an agent is excited to see outcomes that run contrary to its understanding of the world”.

Moreover, we formulated the surprise with the divergence between the true dynamic transition probabilities and the learned transition probabilities, in other words, the relative-entropy of the true transition model with respect to the learned model. Furthermore, we employed a function of the surprise to reshape intrinsic rewards. In the learning process, we used the learned skills to bootstrap the learning of new skills, not only achieving incremental skill learning, but also speeding up model learning and skill learning. In this framework, the incremental skill learning enables agents to solve harder problems with less cognitive effort over time, which is essential in many complex real-world tasks such as rescue robots in cluttered and unknown urban search and rescue (USAR).

The main contributions of this paper are as follows:

1. In the MDP (Markov decision process) task environment, we investigate the incentive mechanisms based on intrinsic motivation, and formalize the surprise-based intrinsic motivation with information theory. The framework leverages the intrinsic motivation both for improving sample efficiency and driving exploration. We demonstrate empirically that the surprise-based intrinsic motivation can result in more efficient exploration.

2. We integrate the surprise-based intrinsic motivation and hierarchical exploration into the framework for the first time. This work not only improves the efficiency of model learning and skill learning, but also improves the ability of incremental learning of more complex skills.

3. The framework of Surprise-HEL incorporates the reward independent incremental learning rules and the technique of alternating model learning and policy update. This work enables the framework to not only handle changing intrinsic rewards and changing models, but to also implement the incremental and developmental learning of models and hierarchical skills.

The rest of this paper is organized as follows. Section 2 describes the related work. Section 3 introduces the necessary background of hierarchical skill learning and information theory. Section 4 describes the framework of surprise-based hierarchical exploration for model and skill learning in detail. Section 5 evaluates the performance in the domain of household robot. Section 6 concludes and discusses future research.

2. Related Work

Intrinsic motivation has many forms of expression including innate preferences, empowerment [15,16,17], novelty [18,19,20], surprise [21,22], predictive confidence, habituation [23], and so on [13,14]. Mohamed et al. [17] used mutual information to form the definition of an internal drive, which is called empowerment. Christopher et al. [12] employed the function of an agent’s confidence to formulate intrinsic rewards. Burda et al. [24] performed the study of purely curiosity-driven learning across 54 standard benchmark environments. Singh et al. [25] adopted an evolutionary perspective to optimize the reward framework, and designed a primary reward function by capturing the pressure, which leads to a notion of intrinsic and extrinsic motivation.

In this work, we used intrinsic motivation to drive exploration and skill learning. Substantial works have been done in this field.

Kulkarni et al. [26] presented a hierarchical-DQN (h-DQN) framework. The framework integrated hierarchical action-value functions and intrinsic motivation. The focus of their work is to make a decision at different temporal scales, but the intrinsic motivation is independent of the current performance of the agent.

McGovern et al. [27] and Menache [28] searched for states that act as bottlenecks to generate skills. Tomar et al. [29] used successor representation to generalize the bottleneck approach to continuous state space. These works mainly focus on building options, rather than improving skill learning performance from intrinsic motivation based exploration.

Achiam et al. [22], Alemi et al. [30], and Klissarov et al. [31] used the approximation of KL-divergence to form intrinsic rewards. Frank et al. [32] empirically demonstrated that artificial curiosity exists in humanoid robots and is capable of exploring the environment through information gain maximization. Fu et al. [33] proposed a discriminator to differentiate states from each other in order to judge whether a state was sufficiently visited. Still et al. [34] defined a curiosity-based, weighted-random exploration mechanism. This exploration mechanism enabled agents to investigate unvisited regions iteratively. Emigh et al. [35] proposed a framework called Divergence-to-Go to quantify the uncertainty of each action-state pair. These works are inefficient in problems with large state and action space due to the underlying models functioning at the level of primitive actions.

In this work, our framework combines the intrinsic motivation-driven exploration and hierarchical exploration to accelerate model learning and hierarchical skill learning. Moreover, the framework motivates the agent to reuse the learned skills to improve its capabilities.

3. Background

3.1. Semi-Markov Decision Process (SMDP)

The semi-Markov decision process (SMDP) is the mathematical model of the hierarchical methods, and provides the theoretical foundation for hierarchical reinforcement learning.

SMDP is a generalization of the Markov decision process (MDP). MDP is defined as a four-tuple $(S,A,R,P)$ where $S$ is the set of states, $A$ is the set of actions, $P$ is the transition function, and $R$ is the reward function. $P(s^{\prime }|s,a)$ is the transition probability of executing the action $a$ in the state $s$ and terminating at the state $s^{\prime }$, and $R(s^{\prime }|s,a)$ is the reward function for transitioning from $s$ to $s^{\prime }$ under the action $a$. $\pi :S\times A\to \left[0,1\right]$ is the stationary stochastic policy. $\pi (a|s)$ is the probability of selecting action $a$ in state $s$.

An SMDP extends an MDP, in which the action can take multiple time steps [36,37]. Let the random variable $k\in \yen +$ denote the number of time steps that action $a$ executes from state $s$ and terminates at state $s^{\prime }$. The state transition probability function can be extended to the joint probability distribution $P(s^{\prime },k|s,a)$, where action $a$ executes from state $s$ and terminates at state $s^{\prime }$ after $k$ time steps. The reward function is changed from $R(s^{\prime }|s,a)$ to $R(s^{\prime },k|s,a)$ accordingly.

3.2. Option

There are three general approaches to formalize hierarchical skills: option [38], MAXQ [36], and HAM [39,40]. In this paper, we worked with the framework of option (temporally extended actions) in skill learning.

An option is a short-term skill that consists of a policy for the state space in a specified region and a termination function for leaving the region. Formally, an option $o$ is defined as a three-tuple $o=\langle I,\pi ,\beta \rangle$, where $I\subseteq S$ is an initiation state set, $\beta :S\to \left[0,1\right]$ is a termination condition function, and $\pi$ is an intra-policy. The action space of the intra-policy contains primitive actions and low-level options. If an option is selected, then this option will be executed according to $\pi$ until its termination.

The low-level options can be regarded as the primitive actions embedded within a high-level option. Moreover, an option can also be regarded as a sub-MDP embedded in a larger MDP. Therefore, all the mechanisms associated with MDP learning also apply to the learning options [41].

3.3. Information Theory

Entropy (Information Entropy) Entropy measures uncertainty between random variables. The entropy $H(X)$ of the discrete random variable $X$ is defined as:

$$H(X)=-{\displaystyle \sum _{x\in \chi }q(x)\log q(x)}$$

(1)

where $\chi$ is the value space of random variable $X$ and $q(x)=\mathrm{Pr}\{X=x\},x\in \chi$ is the probability mass function [42]. Moreover, the entropy of the random variable $X$ can also be interpreted as the expected value of the random variable $\log \frac{1}{q(x)}$:

$$H(X)\stackrel{def}{{\displaystyle =}}H(q)={\mathbb{E}}_q\log \frac{1}{q(x)}$$

(2)

The entropy has the following property: $H(q)\ge 0$ $H(q)=0$ if and only if $q$ is deterministic, and $0\log 0=0$ since $x\log x\to 0$ as $x\to 0$.

Relative-Entropy The relative entropy $\mathrm{D}(q\Vert p)$ is a measure of the inefficiency of assuming that the distribution is $p$ when the true distribution is $q$ [42]. The relative entropy (Kullback–Leibler divergence) between two probability mass functions $q(x)$ and $p(x)$ is defined as

$$\begin{array}{ll}{\mathrm{D}}_{KL}(q||p)& ={\displaystyle \sum _{x\in \chi }^{}q(x)\log \frac{q(x)}{p(x)}}\\ & ={\mathbb{E}}_q\log \frac{q(x)}{p(x)}\end{array}$$

(3)

In the definition, it uses the convention that $0\log \frac{0}{0}=0$ and $0\log \frac{0}{p}=0$ $q\log \frac{q}{0}=\infty$. The relative entropy has the following property: ${\mathrm{D}}_{KL}(q||p)$ is non-negative and ${\mathrm{D}}_{KL}(q||p)=0$ if and only if $p=q$. ${\mathrm{D}}_{KL}(q||p)$ is not in general symmetric: ${\mathrm{D}}_{KL}(q||p)\ne {\mathrm{D}}_{KL}(p||q)$.

4. Surprise-Based Hierarchical Exploration for Model and Skill Learning

4.1. Surprise-Based Intrinsic Motivation

Surprise is one of the intrinsic motivations that arouse interest and drive learning. In this section, we formalized the surprise-based intrinsic motivation with information theory, and used it to weigh the tradeoff between exploration and exploitation.

The surprise is used to measure the uncertainty of a state-action pair. Let surprise denote a measure for the exploration and learning progress when visiting a state-action pair. The surprise changes with the new experience. That is, as the agent continues to explore the environment, the true transition model $Q$ and the learned transition model $P$ in the context of $(s,a)$ gets closer, and the surprise gets smaller.

Exploration and exploitation are two different ways of acting, and is an inherent challenge in reinforcement learning problems [43]. Exploitation is about using the current information to make the best decisions, and exploration is about collecting more information. The tradeoff between exploration and exploitation is that the best long-term policy may include short-term sacrifices to gather enough information to make the best overall decision.

The update step of surprise-based exploration policy makes progress on an approximation to the optimization problem:

$$\max V_{\pi }+\eta \underset{s,a~\pi }{\mathbb{E}}[D_{KL}(Q||P)]$$

(4)

where $V_{\pi }$ is the performance measure, and $\eta >0$ is an exploration–exploitation trade-off coefficient. The left-hand term is to maximize exploitation, and the right-hand term is to drive exploration. In the right-hand term, we formalized surprise in the context of $(s,a)$ with the divergence between the true transition model $Q$ and the learned transition model $P$, in other words, the relative-entropy of $Q$ with respect to $P$.

$$\begin{array}{ll}Surprise& (s,a)=\underset{s,a}{{\displaystyle \Delta }}(Q,P)\\ & =\underset{s,a}{{\mathrm{D}}_{KL}}(Q||P)\\ & =\underset{s^{\prime }~Q(\cdot |s,a)}{{\displaystyle \mathbb{E}}}\log \frac{Q(\mathrm{s},\mathrm{a})}{P(\mathrm{s},\mathrm{a})}\hspace{1em}(P(\mathrm{s},\mathrm{a})\ne 0)\end{array}$$

(5)

Equation (5) shows that in regions of the transition state space, the more times the region is visited, the closer P to Q are in this region. In other words, the surprise of P and Q is more significant in unfamiliar regions, and tends to be 0 in familiar regions.

The exploration incentive in Equations (4) and (5) is intended to minimize the agent’s surprise about its experience. If the surprise is significant, it implies that the agent’s learned model of the environment was insufficient to predict the true environment dynamics and the state transition. The region should be repeatedly investigated until the surprise decreases. If the surprise is low, the learned transition model is approximating the true transition model well. The agent understands the part of the environment well, thus it can move on to explore more uncertain parts to acquire more knowledge and improve its performance.

According to the surprise incentive, we proposed the reshaped form of intrinsic reward as follows:

$$\begin{array}{ll}{r}^{\prime }{}_{\mathrm{int}}(\mathrm{s},\mathrm{a})& =r_{\mathrm{int}}(\mathrm{s},\mathrm{a})-\alpha Surprise(s,a)\\ & =r_{\mathrm{int}}(\mathrm{s},\mathrm{a})-\alpha \left[\underset{s^{\prime }~Q(\cdot |s,a)}{{\displaystyle \mathbb{E}}}\log \frac{Q(\mathrm{s},\mathrm{a})}{P(\mathrm{s},\mathrm{a})}\right]\hspace{1em}(P(\mathrm{s},\mathrm{a})\ne 0)\end{array}$$

(6)

where $\alpha \in (0,1)$ is a positive step-size parameter; $r(\mathrm{s},\mathrm{a})$ is the original reward; and the initial values of intrinsic rewards of all state-action pairs are a constant (e.g., 1). The intrinsic rewards will gradually decrease in the learning process, and the rate of decline is gradually decreasing. Here, the intrinsic rewards are the surprise in the context of $(s,a)$. As above-mentioned, the intrinsic reward in unfamiliar regions will be higher than that of familiar regions. Moreover, after full exploration, the variance between the true model and the learned model is so small that the intrinsic reward will decrease to zero. Essentially, this encourages the agent to select less-selected actions in the current state to go where it is unfamiliar.

However, the intrinsic reward (Equation (6)) cannot be implemented directly, and the true transition model Q is unknown. In order to solve this problem, we introduce the following theorems.

Theorem 1 

(Jensen’s inequality [42]). If $f$ is a convex function and $X$ is a random variable,

$$\mathbb{E}f(X)\ge f(\mathbb{E}X)$$

(7)

Moreover, if $f$ is strictly convex, the equality in Equation (7) implies that $X=\mathbb{E}X$ with probability 1 (i.e., $X$ is a constant).

Theorem 2 

(Convexity of relative entropy [42]). ${\mathrm{D}}_{KL}(Q\Vert P)$ is convex in the pair $(Q,P)$, that is, if $(Q_1,P_1)$ and $(Q_2,P_2)$ are two pairs of probability mass functions, then

$$D(\lambda Q_1+(1-\lambda )Q_2||\lambda P_1+(1-\lambda )P_2)\le \lambda D(Q_1||P_1)+(1-\lambda )D(Q_2||P_2)$$

(8)

for all $0\le \lambda \le 1$.

The intrinsic reward is:

$$\begin{array}{ll}{r}^{\prime }{}_{\mathrm{int}}(\mathrm{s},\mathrm{a})& =r_{\mathrm{int}}(\mathrm{s},\mathrm{a})-\alpha Surprise(s,a)\\ & =r_{\mathrm{int}}(\mathrm{s},\mathrm{a})-\alpha \left[\underset{s^{\prime }~Q(\cdot |s,a)}{{\displaystyle \mathbb{E}}}\log \frac{Q(\mathrm{s},\mathrm{a})}{P(\mathrm{s},\mathrm{a})}\right]\\ & \le r_{\mathrm{int}}(\mathrm{s},\mathrm{a})+\alpha \log \underset{s^{\prime }~Q(\cdot |s,a)}{{\displaystyle \mathbb{E}}}\frac{P(\mathrm{s},\mathrm{a})}{Q(\mathrm{s},\mathrm{a})}\\ & =r_{\mathrm{int}}(\mathrm{s},\mathrm{a})+\alpha \log {\displaystyle \sum _{s^{\prime }~Q(\cdot |s,a)}Q(\mathrm{s},\mathrm{a})\frac{P(\mathrm{s},\mathrm{a})}{Q(\mathrm{s},\mathrm{a})}}\\ & =r_{\mathrm{int}}(\mathrm{s},\mathrm{a})+\alpha \log {\displaystyle \sum _{s^{\prime }~Q(\cdot |s,a)}P(\mathrm{s},\mathrm{a})}{\displaystyle \sum _{s^{\prime }~Q(\cdot |s,a)}P(\mathrm{s},\mathrm{a})}\in (0,1]\end{array}$$

(9)

According to the above equations, the update rule of intrinsic reward is approximated as:

$${r^{\prime }}_{\mathrm{int}}(\mathrm{s},\mathrm{a})\leftarrow r_{\mathrm{int}}(\mathrm{s},\mathrm{a})+\alpha \log {\displaystyle \sum _{s^{\prime }~Q(\cdot |s,a)}P(\mathrm{s},\mathrm{a})}$$

(10)

Ideally, we would like the intrinsic rewards to approach zero in the limit of $P\to Q$, because in this case, the agent should have sufficiently explored the state space. Moreover, with continuous exploration and learning, the value of $\alpha$ is gradually decreasing. Ideally, we could expect that $r_{\mathrm{int}}^{t+1}(\mathrm{s},\mathrm{a})\in (0,r_{\mathrm{int}}^t(\mathrm{s},\mathrm{a}))$. Based on these constraints, the value range of $\alpha$ is limited to:

$$\alpha =\{\begin{array}{cc}0.05& \mathrm{if} r_{\mathrm{int}}(\mathrm{s},\mathrm{a})>0.1-0.05{\displaystyle \sum _{s^{\prime }~Q(\cdot |s,a)}P(\mathrm{s},\mathrm{a})}\\ \frac{{\displaystyle \sum _{s^{\prime }~Q(\cdot |s,a)}P(\mathrm{s},\mathrm{a})}-1}{\log {\displaystyle \sum _{s^{\prime }~Q(\cdot |s,a)}P(\mathrm{s},\mathrm{a})}}\ast r_{\mathrm{int}}^{}(\mathrm{s},\mathrm{a})& otherwise\end{array}$$

(11)

where $r_{int}^{ini}\left(s,a\right)$ is the initial intrinsic reward in the context of $(\mathrm{s},\mathrm{a})$. The update progress of $r_{int}\left(s,a\right)$ is a monotonically decreasing process.

4.2. Reward Independent Hierarchical Skill Model

In this work, we used the framework of option to represent skill. However, the intrinsic rewards are continually changing as the process of exploration and learning in this work, so the traditional option model is not applicable in this case. Therefore, we needed to adopt a reward-independent option model. Currently, Yao et al. [44] have proposed the linear universal option model (UOM) to deal with these problems. In this section, based on the incremental learning rule [41] and UOM [44], we propose reward independent incremental learning rules for hierarchical skill learning to deal with these problems. Moreover, we describe the formal expression of the reward-independent option model.

Formally, an option model of the long-term effect is defined as $\left(R^o,P^o\right)$. This model comprises two parts: the reward model $R^o$ and the transition model $P^o$. $R^o$ and $P^o$ are equivalent to the reward function and transition function in the MDP problem, respectively.

The reward model $R^o$ gives the expected discounted reward, which is received after executing the option o in state s at time t until its termination:

$$R_s^o=\mathbb{E}\{r_{t+1}+\gamma r_{t+2}+\dots +{\gamma }^{k-1}{r}_{t+k}\}$$

(12)

where $k$ is the random execution time for option o from start to termination. $r_{t+k}$ is the reward at the time $t+k$. $\gamma \in [0,1]$ is a discount factor, which affects how much weight it gives to future rewards in the reward model.

The transition model $P^o$ gives the discounted probability of executing option $o$ from state $s$ and terminating at state $s^{\prime }$:

$$P_{{ss}^{\prime }}^o={\displaystyle \sum _{k=1}^{\infty }{\gamma }^{k}p(s^{\prime },k)}$$

(13)

where $p(s^{\prime },k)$ is the probability that option $o$ terminates at state $s^{\prime }$ after $k$ time steps.

The option model has the Bellman equation the same as the value function. For a Markov option $o=\langle I,\pi ,\beta \rangle$, the model is:

$$\begin{array}{ll}{R}_s^o& ={\displaystyle \sum _{a\in A_s}\pi (s,a)}\mathbb{E}\{r_s^a+\gamma (1-\beta (s^{\prime }))R_{s^{\prime }}^o\}\\ & ={\displaystyle \sum _{a\in A_s}\pi (s,a)}{\displaystyle \sum _{s^{\prime }\in S}{p}_{{ss}^{\prime }}^a[r_{{ss}^{\prime }}^a}+\gamma (1-\beta (s^{\prime }))R_{s^{\prime }}^o)]\end{array}$$

(14)

where $\pi (.)$ is a distribution that is consistent with $o=\langle I,\pi ,\beta \rangle$; $s^{\prime }$ is the next state in the context of $(s,a)$; and $r_{{ss}^{\prime }}^a$ is the reward in the context of $(s^{\prime }|s,a)$.

$$\begin{array}{ll}{P}_{{ss}^{\prime }}^o& ={\displaystyle \sum _{a\in A_s}\pi (s,a)}\mathbb{E}\{\gamma [(1-\beta (s_k)P_{s_k{s}^{\prime }}^o+\beta (s_k){\mathbb{I}}_{\{s_k=s^{\prime }\}}]\}\\ & ={\displaystyle \sum _{a\in A_s}\pi (s,a)}{\displaystyle \sum _{s_x\in S}{p}_{s{s}_k}^a\{\gamma [(1-\beta (s_k))p_{s_k{s}^{\prime }}^o+\beta (s_k){\mathbb{I}}_{\{s_k=s^{\prime }\}}]}\}\end{array}$$

(15)

For all $s,s_k,s^{\prime }\in S$, ${\mathbb{I}}_{\{.\}}$ is an indicator function. If its condition is satisfied, it equals 1; otherwise, it equals 0. According to Equations (14) and (15), the temporal-difference update rules are:

$$R_{s_t}^o\leftarrow R_{s_t}^o+\eta \left(r_{t+1}+\gamma (1-\beta (s_{t+1}))R_{s_{t+1}}^o-R_{s_t}^o\right)$$

(16)

$$P_{s_t{s}^{\prime }}^o\leftarrow P_{s_t{s}^{\prime }}^o+\eta \{\gamma [(1-\beta (s_{t+1}))P_{s_{t+1}{s}^{\prime }}^o+\beta (s_{t+1}){\mathbb{I}}_{\{s_{t+1}=s^{\prime }\}}]-P_{s_t{s}^{\prime }}^o\}$$

(17)

where $\eta$ is a positive step-size parameter.

From the perspective of a linear setting, the temporal-difference approximation of $R^o$ is defined as:

$$R_s^o=f^{\top }(U^o\varphi (s))$$

(18)

where $f$ is the least-squares approximation of the expected one-step reward under the option $o$. $U^o$ is the discounted state occupancy function (reward-independent function). $\varphi (s)$ is the feature vector, which maps any state $s\in S$ to its n-dimensional feature representation.

Combining the Bellman equation and the linear setting of the option model, we propose the reward independent incremental learning rules as follows:

$$U_{}^o\leftarrow U_{}^o+\eta \left(\varphi (s_t)+\gamma (1-\beta (s_{t+1}))U_{}^o\varphi (s_{t+1})-U_{}^o\varphi (s_t)\right)\varphi {(s_t)}^{{}^{\top }}$$

(19)

$$P_{}^o\leftarrow P_{}^o+\eta \{\gamma [(1-\beta (s_{t+1}))P_{}^o+\beta (s_{t+1}){\mathbb{I}}_{\{s_{t+1}=s^{\prime }\}}]\varphi (s_{t+1})-P_{}^o\varphi (s_t)\}\varphi {(s_t)}^{{}^{\top }}$$

(20)

The learning of the option model is divided into two parts: the learning of the reward independent function and the learning of the transition mode. In the high-level options, action selection is performed on options rather than primitive actions. Moreover, the modeling of option selection and the update of option models are the same as those of the primitive actions. Thus, we can implement the incremental learning of hierarchical skill models in lifelong learning. In the following section, we detail the specific implementation of the incremental learning framework for hierarchical skills.

4.3. Learning Algorithm

In this work, our framework allows agents to explore the environment and learn skills in a developmental way, and the exploring trajectories can be multi-step. Furthermore, the learned model can be reused to perform complex specific tasks later.

Algorithm 1 gives an overview of the algorithm framework. The algorithm contains two parts. The first part is surprise-based hierarchical exploration and learning (Algorithm 1, lines 2–11). The second part is regularly updating the basic surprise-based exploration policy (Algorithm 1, lines 12–13) as well as the model and the intra-policy of the options (Algorithm 1, lines 14–16).

In the first part, the agent starts with the basic surprise-based hierarchical exploration policy (Algorithm 1, line 2), which is described in detail (Algorithm 2, lines 1–9). In the learning process, the primitive action models have the same representation and learning approach as the option models. The difference is that the primitive action models are only modeled over a single time step, and the termination functions return one in all next states. Then, the framework executes the option $o_t$ and observes the next state (Algorithm 1, line 3). After a round of action selection and execution, the framework updates the intrinsic rewards and the models of primitive actions with current experience (Algorithm 1, line 4). The update methods of intrinsic rewards and models are described in Section 4.1 and Section 4.2, respectively. Next, the framework determines whether to create an option by judging whether the current state is the goal state for the first visit (Algorithm 1, lines 5–8). The new option is made up of a valid goal state, a pseudo-reward function $r_{pse}^o$, and a termination function ${\beta }_o\left(s\right)$. $r_{pse}^o$ and ${\beta }_o(s)$ are all defined as an indicator vector ${\overrightarrow{\mathbb{I}}}_{\{\varphi (s)\in g_i\}}$; if the state $s$ is the goal state of option $o$, then $r_{pse}^o={\beta }_o\left(s\right)=1$, else $r_{pse}^o={\beta }_o\left(s\right)=0$. Finally, the framework updates the executed state stack, the current feature vector and the current time.

Algorithm 1 Surprise-based Hierarchical Exploration for Model and Skill Learning
Input:  current time t, arbitrary initial base policy $\pi$, current executing option stack H, current feature vector $\varphi \left(s_t\right)$, intrinsic reward $r_{int}^a\leftarrow \overrightarrow{1}$, primitive action stack A, base value function ${\theta }_{int}\leftarrow \overrightarrow{0}$, arbitrary initial-policy for any option ${\pi }^o$, option model $\left(U^o,P^o\right)$, option value function ${\theta }^o$, option stack $O$ ($AϵO$), important goal stack G, pseudo reward for any option $r_{pse}^o$, executed state stack D, dataset of transition tuples of the area around the goal sate of option o $D_{area o}$
1:	while$t<t_{max}$do
2:	$o_t\leftarrow$ Surprise-based Hierarchical Exploration $\left(\varphi \left(s_t\right),H,\pi \right)$
3:	$\varphi \left(s_{t+1}\right)\leftarrow$ Execute $\left(\varphi \left(s_t\right),o_t\right)$
4:	update the intrinsic reward and the model of primitive actions $(H,\varphi \left(s_t\right),o_t,\varphi \left(s_{t+1}\right))$
5:	if $\varphi \left(s_t\right)\in g_i$, $\forall g_i\in G$ and $o_i\notin O$ then
6:	$o_i\leftarrow$ create new option $\left(g_i, r_{pse}^o,{\beta }_{o_i}\right)$
7:	$O\leftarrow O{{\displaystyle \cup }}^{ }{o}_i$
8:	end if
9:	update executed state stack $\left(\varphi \left(s_t\right),D\right)$
10:	$\varphi \left(s_t\right)\leftarrow \varphi \left(s_{t+1}\right)$
11:	$t\leftarrow t+1$
12:	Every K time steps do:
13:	$\pi \leftarrow$ Update Surprise-based Exploration Policy $\left({\theta }_{int}, D\right)$
14:	Every T time steps do:
15:	Update Intra-policy of Option $\left(r_{pse}^o,O,D\right)$
16:	Update Option Model $\left(r_{pse}^o,O,D\right)$
17:	end while

In the second part, every $K$ time steps, the framework updates the basic surprise based exploration policy with current experience (Algorithm 1, lines 12–13), which is described in detail (Algorithm 2, lines 11–18). Every $T$ time steps, the framework updates the model and the intra-policy of the learned options with current experience (Algorithm 1, line 14–16). The update method of the intra-policy is described in line 20 to 29 of Algorithm 2, and the update method of the option model is described in Section 4.2. The advantages of alternate learning and updating are to accelerate the learning process of the option policy and to avoid the bad learning effect when the option policy is not perfect.

Algorithm 2 contains specific implementations of some functions in Algorithm 1. The function of Surprise-based Hierarchical Exploration encourages the agent to go where it is unfamiliar to speed up the discovery of new goal states. The agent chooses the primitive action based on $\arg {\max }_a\left[{(r_{\mathrm{int}}^a)}^{{}^{\top }}{U}^a\varphi (s)+\gamma {(P^a\varphi (s))}^{{}^{\top }}{\theta }_{\mathrm{int}}\right]$. Then, it loops over the option stack (exclude primitive actions) to compare with the current option to choose the option that has the max loss value (Algorithm 2, lines 1–9). The function of Update Surprise-based Exploration Policy uses value iteration to update the basic policy of surprise-based exploration. Likewise, the function of Update Intra-policy of Option uses value iteration to update the intra-policy of option.

Algorithm 2 Exploration and Update
1:	function Surprise-based Hierarchical Exploration $\left(\varphi \left(s_t\right),H,\pi \right)$
2:	$o\leftarrow \pi \left(\varphi \left(s_t\right)\right)=argma{x}_a[{(r_{int}^a)}^{\top }{U}^a\varphi (s_t)+\gamma {(P^a\varphi \left(s_t\right))}^{\top }{\theta }_{int}]$
3:	For each $o_i\in \left(O-A\right)$ do
4:	if $-\frac{1}{\left|D_{area }{o}_i\right|}log {\Pi }_{\left(s,a\right)\in D}P\left(s,a\right)>-\frac{1}{\left|D_{area }o\right|}log{\Pi }_{\left(s,a\right)\in \mathrm{D}}P\left(s,a\right)$ then
5:	$o=o_i$
6:	end if
7:	end for
8:	Return o
9:	end function
10:
11:	function Update Surprise-based Exploration Policy $\left({\theta }_{int},D\right)$
12:	for N iterations do
13:	for each $\varphi \left(s\right)\in \mathrm{D}$ do
14:	${\theta }_{int}\leftarrow {\theta }_{int}+\alpha ma{x}_a[{\left(r_{int}^a\right)}^{\top }{U}^a\varphi \left(s\right)+\gamma {\left(P^a\varphi \left(s\right)\right)}^{\top }{\theta }_{int}-\varphi \left(s{)}^{\top }{\theta }_{int}\right]\varphi \left(s\right)$
15:	end for
16:	end for
17:	$\pi \left(s\right)\leftarrow arg ma{x}_a[{\left(r_{int}^a\right)}^{\top }{U}^a\varphi \left(s\right)+\gamma {\left(P^a\varphi \left(s\right)\right)}^{\top }{\theta }_{int}]$
18:	end function
19:
20:	function Update Intra-policy of Option $\left(r_{pse}^o,O,D\right)$
21:	For each $o_i\in \left(O-A\right)$ do
22:	for N iterations do
23:	for each $s\in D{{\displaystyle \cup }}^{ }{I}^{o_i}$ do
24:	${\theta }^{o_i}\leftarrow {\theta }^{o_i}+\alpha [{\left(r_{pse}^o\right)}^{\top }{U}^o\varphi \left(s\right)+\gamma {\left(P^o\varphi \left(s\right)\right)}^{\top }{\theta }^{o_i}-\varphi \left(s{)}^{\top }{\theta }^{o_i}\right]\varphi \left(s\right)$
25:	end for
26:	end for
27:	${\pi }^{o_i}\left(s\right)\leftarrow \arg ma{x}_o[{\left(r_{pse}^{o_i}\right)}^{\top }{U}^o\varphi \left(s\right)+\gamma {\left(P^o\varphi \left(s\right)\right)}^{\top }{\theta }^{o_i}]$
28:	end for
29:	end function

4.4. Working Example

In this section, we describe a simple working example to make each definition clearer. Figure 1 is a simplified version of our experimental environment. The robot can be anywhere in the room. We assume that the robot starts at the position $s$. $s1$ and $s2$ are the goal positions. The purpose of the robot is to explore the entire environment aimlessly based on intrinsic motivation and learn the skills to reach the goal positions from any position in the room. The working example is described as follows.

Step 1: The robot explores the environment according to the surprise based hierarchical exploration policy (Algorithm 1, line 2; Algorithm 2, lines 1–9), and continuously updates the transition model and intrinsic reward of primitive actions (Algorithm 1, line 4).

Step 2: When the robot first reaches the position $s1$, it finds that $s1$ is a goal position whose associated option has not yet been created. Then, the robot creates a new option $o_{s1}$ for reaching the position $s1$, and adds the new option to the option stack. The new option is made up of a valid goal state $s1$, a pseudo-reward function $r_{pse}^o$, and a termination function ${\beta }_o(s)$.

Step 3: During the learning and exploration process, the framework updates the surprise based exploration policy every $K$ time steps as follows (Section 4.3, Algorithm 2, line 17).

$$\pi (s)\leftarrow \arg {\max }_a\left[{(r_{\mathrm{int}}^a)}^{{}^{\top }}{U}^a\varphi (s)+\gamma {(P^a\varphi (s))}^{{}^{\top }}{\theta }_{\mathrm{int}}\right]$$

(21)

where $r_{\mathrm{int}}^a$ is the intrinsic reward, and is reshaped by surprise, which is described in Equation (10) (Section 4.1). $a$ is a primitive action or an option.

Step 4: Every $T$ time steps, the framework optimizes the model and the intra-policy of the learned options with simulation. The framework updates the option model according to Equations (19) and (20) (Section 4.2), and updates the intra-policy of the learned options as follows (Section 4.3, Algorithm 2, line 27). Figure 2 shows that the intra-policy of option $o1$ changes from incomplete (Figure 2a) to optimal (Figure 2b) after several optimizations.

$${\pi }^{o_i}(s)\leftarrow \arg {\max }_o\left[{(r_{pse}^{o_i})}^{{}^{\top }}{U}^o\varphi (s)+\gamma {(P^o\varphi (s))}^{{}^{\top }}{\theta }^{o_i}\right]$$

(22)

Step 5: The robot uses the intra-policy of the learned option $o_{s1}$ to reach the unfamiliar area more quickly, which is hierarchical exploration. When the robot first encounters the position s2, it constructs the new option $o_{s2}$, and it can use the intra-policy of option $o_{s1}$ to learn the intra-policy of option $o_{s2}$ faster. For example, the intra-policy of option $o_{s2}$ from s to s2 only needs to learn from s1 to s2 because it can use the existing intra-policy of option $o_{s1}$ from s to s1.

Step 6: After multiple time steps, the learned model will get closer to the true model, and the intrinsic rewards will tend to zero in the limit of P$\to Q$. This means that the robot has sufficiently explored the state space. Moreover, the learned options underlie both the ability to choose to spend effort and time to specialize at particular tasks, and the ability to collect and exploit previous experience to be capable of solving harder problems over time with less cognitive effort.

5. Experiments

5.1. Household Robot Pickup and Place Domain

We tested the performance of our framework in the domain of Household Robot Pickup and Place. The problem is modeled as the robot picks up an object and places it at designated positions in the discrete rooms (Figure 3). The test domain is a variant of the hierarchical reinforcement learning household robot [45]. The experiments are designed with increasing state sizes to demonstrate that the proposed methods are scalable to handle large problems.

We preset ten goal positions of g1~g10. The destination position and the initial position of the object were randomly selected from the ten positions. The destination position and the initial position were two different positions. For example, if the destination position is g1, then the initial position of the object should be selected from g2~g10. Moreover, there are also many other notable positions (D1~D10). The goal positions and notable positions were used as sub-goals to create corresponding options (e.g., option $o_{D1}$ is defined as the transition from room 4 to room 1, and option $o_{g1}$ is defined as the transition from any position in room 1 to position g1).

In this work, we split the experiment into separate phases. In the first phase, the robot first explores the environment without a specific task, and it learns the skills in a developmental way. In the course of exploration and learning, the surprise-based hierarchical exploration can improve the exploration efficiency and speed up the model learning and skill learning. The robot also continues to refine the models and intra-policies of skills through simulation. In the second phase, the robot leverages the learned skills to perform the specific task (Robot Pickup and Place) in the framework of hierarchical MCTS (Monte Carlo Tree Search).

5.2. Performance Evaluation

The surprise based intrinsic motivation is defined as the doing of an activity for its own sake rather than to directly solve some specific problems. Therefore, evaluating the benefits of surprised-base hierarchical exploration is not as simple as evaluating the performance of a standard reinforcement learning in a particular task. Therefore, we will evaluate the performance of our framework with model accuracy, loss value, the number of visited states, and the task performance in specific task with the learned model and options.

A typical loss is:

$$ℒ\left(P;D\right)=-\frac{1}{\left|D\right|}\log {\displaystyle \prod }_{(s,a,s^{\prime })\in D_{}}P\left(s^{\prime }|s,a\right)$$

(23)

where $P$ denotes the learned transition model, which approximates the true model $Q$. $D$ is the dataset of transition tuples from the environment. In general, learning $P(s,a)$ implies minimizing the loss $ℒ\left(P\left(s,a\right);D\right)$. The closer $P$ and $Q$ are, the smaller the loss.

The model accuracy is defined as:

$$\overline{P}\left(D\right)=\frac{1}{\left|D\right|}{\displaystyle \sum _{(s,a,s^{\prime })\in D}P\left(s^{\prime }|s,a\right)}$$

(24)

To compare the performance of our framework with the performance of other methods, we used three different exploration policies, and these exploration policies guide behavior as the agent learns skills in the domain. The comparison will help us verify whether the proposed framework can accelerate exploration and skill learning.

Random exploration. A random exploration chooses a random action from the sets of primitive actions and options. It executes each action or option to completion before choosing another one. It is the baseline approach.

Exploration with Exemplar Models (EX²) [33]. We use the heuristic bonus they used in their experiments to reshape the intrinsic rewards to drive exploration.

Surprise-HEL. This method employs surprise-base hierarchical exploration policy. The combination of surprise and hierarchical exploration allows the agent to reach the areas with more information faster.

5.3. Results

In this section, we present the evaluation results for Household Robot Pickup and Place domain. In the following content, we first show the performance of model learning and skill learning where there is no external reward. Then, we applied the learned model in the first step to a specific task to evaluate the task performance.

5.3.1. Performance of Model and Skill Learning

Figure 4a,c,e show the performance of model learning and skill learning in the domain of 10 × 10. Similarly, Figure 4b,d,f show the performance of model learning and skill learning in the domain of 20–20. In this experiment, all of the results were averaged over 100 runs.

From Figure 4a–d, we can see that compared with other methods, Surprise-HEL learns more accurate models when they all take the same number of steps, and can converge to the true transition model faster. Random exploration failed in obtaining convergence in both domains.

While learning relatively accurate models with finite steps, it is more important for the algorithm to explore unfamiliar and useful parts of the domain. Figure 4e,f show the average number of visited states of the three methods for both the 10 × 10 domain and 20 × 20 domain, which demonstrate the performance of the skill learning of different algorithms in these domains. Surprise-HEL performed better than the other two algorithms, and the random exploration failed in obtaining convergence in both domains. The reason is that the Surprise-HEL leverages the learned options and the surprise-based intrinsic motivation to speed up exploring unfamiliar areas. As a result, the more states the robot visits, the more options will be discovered to accelerate skill learning.

Taking the same number of steps in the domain of 10 × 10, it shows that in the term of model learning, compared with EX² and random exploration, the maximum differences of model accuracy were 4.73 and 19.64, and the maximum increases were 15.95% and 37.00%, respectively. Similarly, in the domain of 20 × 20, the maximum differences were 7.38 and 23.45, and the maximum increases were 18.80% and 69.17%, respectively.

In the term of skill learning, compared with EX² and random exploration and taking the same number of steps, the maximum differences of the visited states were 85.46 and 193.53, and the maximum increases were 38.40% and 105.37%, respectively. Similarly, in the domain of 20 × 20, the maximum differences were 361.90 and 790.83, and the maximum increases were 42.57% and 150.25%, respectively. The specific analysis results are shown in

Table 1. The performance comparison of model learning and skill learning under the same steps.

The comparison in model learning
Algorithm	10 × 10		20 × 20
	Difference	Increase	Difference	Increase
EX2	4.72927569301687	15.9545090896041%	7.386379431	18.7976065%
random	19.6397000224526	37.0010595733331%	23.44520569	69.1739%
The comparison in skill learning
Algorithm	10 × 10		20 ×20
	Difference	Increase	Difference	Increase
EX2	85.456489458935	38.4035087719295%	361.899865560966	42.5681606254505%
random	193.533333333334	105.372050816698%	790.833333333327	150.248459550786%

.

This result shows that the Surprise-HEL consistently outperformed other methods. Moreover, the performance improvements were much more dramatic in both model learning and skill learning in the larger domain. This further demonstrates the efficiency of our framework.

5.3.2. Performance of Specific Task

Next, we applied the learned model and skills to a specific task. The specific task was defined as the robot picks up the object in position g5 and places it in position g9. The initial position of the robot was (1, 1). There were three state variables: the position of the robot, the position of the object, and the position of the destination. There were four primitive actions: move up, move down, move left, and move right. Each primitive action had a reward of −1. Reaching the object’s initial position and destination position both have a reward of 10. The maximal planning horizon was 400.

The performance was evaluated using the average cumulative return in terms of the number of simulations. Each data point was averaged over 100 runs in terms of the number of simulations.

Figure 5 shows the cumulative reward received by option based hierarchical MCTS and flat MCTS over 2¹⁸ simulations of the task. It shows that the option based hierarchical MCTS significantly outperformed the flat MCTS with limited computational resources. The option based hierarchical MCTS could find the optimal policy faster than flat MCTS. The reason is that the feature of option based hierarchical search can significantly reduce computational cost while speeding up learning and planning. Moreover, the rollouts of option based hierarchical MCTS sometimes terminate early, since the learned options help the rollouts to reach more useful parts of the state regions. Therefore, the option based MCTS can find the strategy to complete the task faster. Intuitively, the effect of option based hierarchical MCTS should become more significant in larger domains with longer planning horizons.

Additionally, the primary purpose of our work was to apply this framework to the domain of robot urban search and rescue (USAR) environments in a cluttered environment. However, the state space of the real world is continuous, dynamic, partially observable, and there are many uncertain factors. Applying our framework to this domain is a complex project, and our future work is to solve these problems.

6. Conclusions

In this work, we proposed the framework of surprise-based hierarchical exploration for model and skill learning (Surprise-HEL). This framework has three main features: surprise-based hierarchical exploration, reward independent incremental learning rules, and the technique of alternating model learning and policy update. The nature of the framework implements the incremental and developmental learning of models and hierarchical skills. In the experiment of household robot domain, we empirically showed that Surprise-HEL performed much better than other algorithms both for model learning and skill learning, and it performed better in large-scale problems. Moreover, we applied the learned model to a specific task to evaluate the task performance, and the results showed that the skill based hierarchical planning significantly outperformed the flat planning with limited computational resources. In future work, we plan to apply this method to complex real-world application scenarios such as rescue robots in cluttered and unknown urban search and rescue (USAR). We expect that our method can accelerate the robot’s exploration and learning in a complex and unknown environment, and improve the robot’s search and rescue capabilities based on the learned skills.