Latent Structure Matching for Knowledge Transfer in Reinforcement Learning

Abstract

Reinforcement learning algorithms usually require a large number of empirical samples and give rise to a slow convergence in practical applications. One solution is to introduce transfer learning: Knowledge from well-learned source tasks can be reused to reduce sample request and accelerate the learning of target tasks. However, if an unmatched source task is selected, it will slow down or even disrupt the learning procedure. Therefore, it is very important for knowledge transfer to select appropriate source tasks that have a high degree of matching with target tasks. In this paper, a novel task matching algorithm is proposed to derive the latent structures of value functions of tasks, and align the structures for similarity estimation. Through the latent structure matching, the highly-matched source tasks are selected effectively, from which knowledge is then transferred to give action advice, and improve exploration strategies of the target tasks. Experiments are conducted on the simulated navigation environment and the mountain car environment. The results illustrate the significant performance gain of the improved exploration strategy, compared with traditional $ϵ$-greedy exploration strategy. A theoretical proof is also given to verify the improvement of the exploration strategy based on latent structure matching.

1. Introduction

Reinforcement learning (RL) is where an agent guides its actions based on the rewards obtained from the trial-and-error interaction with the environment [1,2]. The ultimate goal of RL is to achieve a mapping from states to actions with maximum return [3]. In recent years, reinforcement learning has been successfully applied to some aspects of life such as medical care [4], finance [5], and transportation [6]. However, in RL, a small change on the problem configuration usually leads to a complete new training, which inevitably brings a slow convergence and the growing requirement of a large amount of empirical samples. The introduction of transfer learning [7] in RL provides a promising solution to this problem.

Transfer learning guides the transfer of useful knowledge from well-learned source tasks to target tasks [8]. In RL, the knowledge obtained from previous situations can be reused as heuristics to achieve effective knowledge transfer, thus speeding up the learning procedure in new situations and reducing sample request [3]; moreover, knowledge transfer is able to mitigate much the issue caused by a change on the problem configuration as mentioned above.

However, if an inappropriate source task is selected in knowledge transfer, it will lead to negative transfer [9], which will slow down or even disrupt the learning of a new target task. Therefore, it is very important for knowledge transfer to select appropriate source tasks that have a high degree of matching with target tasks.

The similarity estimation of tasks is the main way to select matched source tasks in the existing works on knowledge transfer for RL. The works proposed at the beginning are semi-automatic, because they require human intervention for similarity estimation. A typical work [10] defined a one-to-one mapping between states and actions to estimate similarity. In the subsequent work [11], a many-to-one mapping of relevant states and actions was defined. These semi-automatic works require manual parameterization of defined mappings; nevertheless, the optimal values of the parameters are not easy to reach by experience or intuition [12]. In order to overcome this problem, some mechanisms [13,14] to select optimal mappings were developed. However, human intervention is still needed in these works [12].

Later, in order to solve the problems in the earlier works mentioned above, some automatic similarity estimation works [15,16,17,18,19,20,21] are presented. These works made use data-driven similarity metrics, including the Markov decision process (MDP) similarity metric [20], the Hausdorff metric [21], and the Kantorovich metric [21]. These works are highly interpretative, but computationally expensive and have trouble when handling situations involving a large number of tasks [19].

Recently, clustering algorithms [22] were used in some works to tackle large number of tasks. In these works, the clustering of policies, value functions, rewards, and dynamics of tasks, were modeled as random process to estimate the similarity [23,24]. However, these clustering-based works suffer from the bias from the modeling for clustering, and are sensitive to noise and variation in policies, dynamics, and value functions of tasks, resulting in the inaccurate estimation of task similarity.

The classification and characteristics of existing similarity estimation methods mentioned above are summarized in Table 1.

Table 1. Classification of similarity metrics.

Categories of Similarity Metrics	Features	Related References
Semi-automatic	Depend on the experience of experts	[10,11,13,14]
Automatic(non-clustering-based)	Highly interpretative, but computationally expensive	[15,16,17,18,19,20,21]
Automatic(clustering-based)	Can tackle large number of tasks, but sensitive to noise	[23,24]

Classification and characteristics of existing similarity estimation methods.

To address the limitation of existing works, we propose a novel task matching algorithm, namely latent structure matching (LSM), which uses low-dimensional embedding to extract latent essential factors and relationships of tasks. Furthermore, through low-dimensional alignment, LSM effectively matches tasks in large number, and estimates their similarity, thus finding highly-matched source tasks. Based on the matched source tasks, useful exploratory advice is formed to improve exploration strategy, and to accelerate the learning of target tasks. The experiments are conducted on the simulated maze navigation environment and the mountain car environment. The maze navigation environment is an environment with discrete state and action space, and the mountain car environment is an environment with continuous state space. The results of experiments illustrate the significant performance improvement of RL agents, when taking improved exploration strategies with the action advice from the matched source tasks selected effectively by LSM.

Our main contributions in this paper can be summed up as follows:

(i)

A novel task matching algorithm, namely LSM, is proposed for RL to derive latent structures of value functions of tasks, and align the structures for similarity estimation. Structural alignment permits efficient matching of large number of tasks, and locating the correspondences of tasks.

(ii)

Based on LSM, we present an improved exploration strategy, that is built on the knowledge obtained from the highly-matched source task. This improved strategy reduces random exploration in value function space of tasks, thus effectively improving the performance of RL agents.

(iii)

A theoretical proof is given to verify the improvement of exploration strategy with the latent structure matching-based knowledge transfer (Please see Appendix B).

In the remainder of this paper, related work is first discussed in Section 2. The proposed method is introduced in Section 3. The detailed experimental procedures and experimental results are presented in Section 4. Finally, a brief conclusion is drawn in Section 5.

2. Related Work

2.1. Knowledge Transfer in RL

The sample complexity and learning inefficiency of the RL algorithms are two of the most critical aspects that obstruct its feasibility in practical applications [25]. Transfer learning is introduced to address these problems.

To perform effective knowledge transfer in RL, it is of great importance to select appropriate source tasks that are highly-matched with the target task. In most of the existing works, similarity estimation is the main way of selecting a matched source task. Early works were semi-automatic, as they conducted the similarity estimation of tasks according to human experience or intuition. Taylor and Whiteson [26] defined one-to-one inter-task mapping (TVITM) as an action in one task and its image in another task being similar with each other. The similarity was loosely defined so that it depended on how the action affected the agent’s state and what reward was received. TVITM was subsequently used by Taylor and Stone [10] for knowledge transfer in RL to speed up learning. A many-to-one mapping for knowledge transfer in RL was presented in [11], which used a linear combination of relevant states and actions to initialize learning. However, it needed humans to provide the parameters of the linear combination, and the optimal values of the parameters were not easy to reach [12]. To avoid this, some researchers developed mechanisms for optimal mapping selection. Anestis and Ioannis [13] proposed two algorithms to select the optimal mapping for both model-based and model-free RLs respectively. In another of their works [14], they proposed a method which autonomously selected mappings from a set of all possible inter-task mappings.

In order to learn similarity of tasks without human intervention, some automatic works have been proposed. Ferns [15] measured the distance between Markov decision processes (MDPs) by constructing bisimulation metrics on rewards and probability functions. Taylor [16] gave a method called “modeling approximate state transitions by exploiting regression (MASTER)”, which automatically learned a mapping from one task to another based on evidences gathered from task environments. A method using neural work to automatically map actions in one domain to actions in the other domain was proposed in [17]. Carroll and Seppi [18] defined similarity in terms of tasks, and proposed several possible task similarity measures based on the transfer time, policy overlap, Q-values, and reward structure respectively to measure the similarity between tasks. Teng [19] proposed to use self-organization neural networks to make the effective use of domain knowledge to reduce model complexity in RL. Through minimizing the reconstruction error of a restricted Boltzmann machine simulating the behavioral dynamics of two compared Markov decision processes, Ammar and Eaton [20] gave a data-driven automated Markov decision process similarity metric. Song and Gao [21] introduced two metrics for measuring the distance between finite Markov decision processes: The Hausdorff metric measuring the distance between two subsets of a metric space, and the Kantorovich metric measuring the distance between probabilistic distributions. These works are from a purely computational perspective, so they are computationally intensive, and cannot handle scenarios with a large number of source tasks [19].

To solve the issue raised when handling large number of tasks, recent works used the clustering to learn the similarity automatically. Li and Liao [23] used the intrinsic clustering property of the Dirichlet process to impose sharing of episodes among similar tasks, which effectively reduced the number of episodes required for learning a good policy in each task, when data sharing was appropriate. Lazaric and Ghavamzadeh [27] used a Gaussian process hierarchical BNP model to cluster different classes of source MDPs in sequential batch mode. Wilson and Fern [28] classified different Markov decision processes by clustering the dynamics and rewards which were drawn from the Dirichlet process mixture models. A flexible Bayesian nonparametric model was exploited to cluster different classes of source Markov decision processes by the dependent Dirichlet process-based clusters of Gaussian process in [24]. However, these clustering-based methods need to use methods such as Bayesian model to simulate distributions, bias will arise during the simulation process, and the final results of the clustering-based methods are based on the simulated distributions, so the effects of these methods are easily affected by the simulation bias. Further, these methods suffer from the sensitivity to the noise and variation in the policies, dynamics, and value functions of source tasks.

We follow the automatic work line to select the highly-matched source tasks by LSM. LSM does not require the distribution model assumption for the task value space, thus reducing the bias from the assumption. Also, latent structure deriving in LSM benefits the suppression of noise and outliers. Furthermore, structural alignment of LSM guarantees the efficiency of handling a large number of tasks.

2.2. Low Rank Embedding

The algorithm we use to derive latent structures of value function space falls in the category of low rank embedding (LRE) [29]. Low rank embedding is a two-part algorithm for solving embedding in low-dimensional space by using the low rank matrix approximation method. Now given a data set X, at the beginning, LRE minimizes the following loss function so as to calculate the reconstruction coefficient matrix $\widehat{X}$:

$$\frac{1}{2}\parallel X-X\widehat{X}{\parallel }_F^2+\lambda {\parallel \widehat{X}\parallel }_{*},$$

(1)

where $\lambda >0$, ${\parallel X\parallel }_F=\sqrt{{\sum }_i{\sum }_j{\left|X_{i,j}\right|}^2}$ is the Frobenius norm. As for singular values ${\sigma }_i$, ${\parallel X\parallel }_{*}={\sum }_i{\sigma }_i\left(X\right)$ is the spectral norm. LRE uses the alternating direction method of multipliers (ADMM) algorithm [30] to minimize (1) to calculate the reconstruction coefficient matrix $\widehat{X}$, and then minimizes the reconstruction error of X in low-dimensional space to obtain the embedding F. The reconstruction error function is as follows:

$$\frac{1}{2}{\parallel F-F\widehat{X}\parallel }_F^{2}s.t.F^{T}F=\mathbb{I},$$

(2)

where $\mathbb{I}$ is the identity matrix, and the constraint $F^{T}F=\mathbb{I}$ can remove arbitrary scaling factors in the low-dimensional embedding and can ensure that the minimization problem of (2) is a well-posed problem. In [31], it is shown that (2) can be minimized to obtain the embedding F by calculating the d smallest non-zero eigenvectors of the Gram matrix ${(\mathbb{I}-\widehat{X})}^T(\mathbb{I}-\widehat{X})$.

To estimate efficiently the similarity between source and target tasks, we modify low rank embedding, and utilize the modified algorithm in RL (please see Section 3.1). To the best of our knowledge, our approach is a novel attempt to use LRE in knowledge transfer for RL.

3. Method

The proposed method is divided into two stages. Figure 1 shows an overview of the two-staged method composed of latent structure matching and value function transfer.

Figure 1. Overview of the two-staged method composed of latent structure matching and value function transfer. In the first stage, latent structure matching (LSM) is used to measure the similarity between tasks, so as to find the highly-matched source task for the target task. In the second stage, value function transfer is used to improve the agent’s exploration strategy, thereby improving the agent’s performance.

As shown in Figure 1, in the first stage, latent structures are derived to estimate the similarity between tasks and find the highly-matched source task for a new target task. In the second stage, value function transfer is conducted to improve the exploration strategy of agents by using action advice extracted from the highly-matched source task. The details of each stage will be given in the following subsections.

3.1. Latent Structure Matching (LSM)

The algorithmic basis of LSM is manifold alignment [32], through which LSM derives latent structures of tasks and finds matched source task for a new target task. LSM is actually one of the variants of LRE. Like normal LRE, it firstly calculates a reconstruction coefficient matrix for source and target tasks, and then minimizes the reconstruction error in low-dimensional space. However, compared with normal LRE, several modifications are made in the reconstruction error function and reconstruction coefficient matrix for efficient estimation of similarity between source and target tasks.

Given action value functions learned for S source tasks, we arrange all these functions into a vector $Q={\left[Q_l^{\left(s\right)}\right]}_{1\times L}$, where $Q_l^{\left(s\right)}$ is the $l^{th}$ action value function for the $s^{th}$ source task. Similarly, all of the action value functions learned for T target tasks are also arranged into the vector $\mathcal{Q}={\left[{\mathcal{Q}}_{\gamma }^{\left(t\right)}\right]}_{1\times \Gamma }$, where ${\mathcal{Q}}_{\gamma }^{\left(t\right)}$ is the ${\gamma }^{th}$ action value function for the $t^{th}$ target task.

The Equation (2) can be used to derive source and target tasks’ latent structures, namely the low-dimensional embeddings of action value functions Q and $\mathcal{Q}$. To match the embeddings in low dimension, we add an alignment item and obtain our error function:

$$S\left(\mathrm{\Phi }\right)={(1-\mu )\parallel \mathrm{\Phi }-B\mathrm{\Phi }\parallel }_F^2+\mu \sum _{i,j=1}^m{\parallel {\mathrm{\Phi }}_i-{\mathrm{\Phi }}_j\parallel }^2{C}_{i,j},$$

(3)

where $\mu \in [0,1]$ is a weight parameter. The first item of (3) represents the reconstruction error, and it is consistent with the right side of (2); the second item is added to represent the alignment error. $\mathrm{\Phi }\in {\mathbb{R}}^{m\times d}$, $B\in {\mathbb{R}}^{m\times m}$, and $C\in {\mathbb{R}}^{m\times m}$ are all block matrices:

$$\mathrm{\Phi }=\left[\begin{array}{c}E\\ \mathcal{E}\end{array}\right],B=\left[\begin{array}{cc}\widehat{Q}& 0\\ 0& \widehat{\mathcal{Q}}\end{array}\right],C=\left[\begin{array}{cc}0& C^{(Q,\mathcal{Q})}\\ C^{(\mathcal{Q},Q)}& 0\end{array}\right],$$

where $m=L+\Gamma$ and d is the dimension of low-dimensional space. E and $\mathcal{E}$ are the embeddings respectively for $Q,\mathcal{Q}$. $C^{(Q,\mathcal{Q})}$ is the proximity matrix, which can be obtained by using cosine similarity metric. $\widehat{Q}$ and $\widehat{\mathcal{Q}}$ are the reconstruction coefficient matrices of Q and $\mathcal{Q}$, respectively, which can be solved by minimizing (1). However, the minimization of (1) in LRE requires frequent iterative calculation. For this, we use the closed-form solution of subspace estimation [33] to calculate the reconstruction coefficient matrix, thereby reducing the calculation. The specific procedure is described in Appendix A.

The error function in (3) can be reduced to a sum of matrix traces:

$$\begin{array}{cc}\hfill S\left(\mathrm{\Phi }\right)& =(1-\mu )tr\left({(\mathrm{\Phi }-B\mathrm{\Phi })}^T(\mathrm{\Phi }-B\mathrm{\Phi })\right)+\mu \sum _{z=1}^d\sum _{i,j=1}^m{\parallel {\mathrm{\Phi }}_{i,z}-{\mathrm{\Phi }}_{j,z}\parallel }_2^2{C}_{i,j}\hfill \\ \hfill & =(1-\mu )tr\left({\left((\mathbb{I}-B)\mathrm{\Phi }\right)}^T(\mathbb{I}-B)\mathrm{\Phi }\right)+2\mu \sum _{z=1}^d{\mathrm{\Phi }}_{.,z}^{T}L{\mathrm{\Phi }}_{.,z}\hfill \\ \hfill & =(1-\mu )tr\left({\mathrm{\Phi }}^T{(\mathbb{I}-B)}^T(\mathbb{I}-B)\mathrm{\Phi }\right)+2\mu tr\left({\mathrm{\Phi }}^{T}L\mathrm{\Phi }\right).\hfill \end{array}$$

(4)

Similar to locally linear embedding [34] and LRE, we introduce the constraint ${\mathrm{\Phi }}^T\mathrm{\Phi }=\mathbb{I}$ to ensure that the minimization problem of error function S is a well-posed problem. Thus, we have

$$\begin{array}{cc}\hfill a& rgmi{n}_{\mathrm{\Phi }:{\mathrm{\Phi }}^T\mathrm{\Phi }=\mathbb{I}}S=argmi{n}_{\mathrm{\Phi }:{\mathrm{\Phi }}^T\mathrm{\Phi }=\mathbb{I}}(1-\mu )tr\left({\mathrm{\Phi }}^{T}A\mathrm{\Phi }\right)+2\mu tr\left({\mathrm{\Phi }}^{T}L\mathrm{\Phi }\right),\hfill \end{array}$$

(5)

where $A={(\mathbb{I}-B)}^T(\mathbb{I}-B)$, and L is the combinatorial graph Laplacian $L=D-C$, where D is the diagonal matrix of row sums $D_{i,i}={\sum }_j{C}_{i,j}$. To construct a loss function from (5), a Lagrange multiplier $\delta$ is introduced to the right side of (5):

$$\mathcal{L}(\mathrm{\Phi },\delta )=(1-\mu )tr\left({\mathrm{\Phi }}^{T}A\mathrm{\Phi }\right)+2\mu tr\left({\mathrm{\Phi }}^{T}L\mathrm{\Phi }\right)+⟨\delta ,{\mathrm{\Phi }}^T\mathrm{\Phi }-\mathbb{I}⟩.$$

(6)

To minimize the loss function (6), the roots for the partial derivatives of $\mathcal{L}$ for $\mathrm{\Phi }$ and $\delta$ are calculated as:

$$\left\{\begin{array}{c}\frac{\partial \mathcal{L}}{\partial \mathrm{\Phi }}=(1-\mu )A\mathrm{\Phi }+2\mu L\mathrm{\Phi }-\delta \mathrm{\Phi }=0\hfill \\ \frac{\partial \mathcal{L}}{\partial \delta }={\mathrm{\Phi }}^T\mathrm{\Phi }-\mathbb{I}=0.\hfill \end{array}\right.$$

(7)

Therefore, to optimize (5) to obtain embedding $\mathrm{\Phi }$, we need to calculate the d smallest non-zero eigenvectors of matrix $\left(\right(1-\mu )A+2\mu L)$. After the embedding $\mathrm{\Phi }$ is obtained, the distance between E and $\mathcal{E}$, that represents the similarity between source and target task set, can be estimated. Finally, the highly-matched source task for a target task can be found. The whole procedure of LSM is fully described in pseudocode Algorithm 1.

Algorithm 1 Latent structure matching

Input: source action value function Q; target action value function $\mathcal{Q}$; embedding dimension d; weight parameter $\mu$ Output: embedding matrix E of source task; embedding matrix $\mathcal{E}$ of target task. 1: Compute the reconstruction coefficient matrices as described in Appendix A: 2: Build the block matrices $\mathrm{\Phi }$, B and C. 3: Construct the error function (3), namely $S\left(\mathrm{\Phi }\right)={(1-\mu )\parallel \mathrm{\Phi }-B\mathrm{\Phi }\parallel }_F^2+\mu {\sum }_{i,j=1}^m{\parallel {\mathrm{\Phi }}_i-{\mathrm{\Phi }}_j\parallel }^2{C}_{i,j}$ 4: Minimize the loss function (6): $\mathcal{L}=(1-\mu )tr\left({\mathrm{\Phi }}^{T}A\mathrm{\Phi }\right)+2\mu tr\left({\mathrm{\Phi }}^{T}L\mathrm{\Phi }\right)+⟨\delta ,{\mathrm{\Phi }}^T\mathrm{\Phi }-\mathbb{I}⟩$ 5: The embedding matrices E and $\mathcal{E}$ can be obtained by calculating the d smallest non-zero eigenvectors of matrix $(1-\mu )A+2\mu L$.

3.2. Value Function Transfer

After finding the highly-matched source task, the learnt action value functions of the matched source task can be directly transferred to the target task. However, if there is not enough similarity between source tasks and target tasks, it will result in the risk of negative transfer due to direct transfer. So using the action value functions of the matched source task to guide the agent in action exploration may be a safe program.

The most commonly used exploration strategy is the $ϵ$-greedy exploration strategy [35], whose development purpose is to strike a balance between exploration and exploitation. This balance determines whether one should explore the invisible parts of the space or use the knowledge that has been acquired during the learning process. Specifically, the $ϵ$-greedy exploration strategy performs exploitation with the probability $ϵ$ by selecting a known action that maximizes the value function of a target task, and performs exploration with the probability 1-$ϵ$ by randomly selecting an unknown action.

By using the knowledge from the source tasks, the $ϵ$-greedy exploration strategy is modified as shown in Algorithm 2. The modified strategy, which we call an LSM-based exploration strategy, performs exploitation as normal $ϵ$-greedy exploration strategy, but performs exploration with a probability of 1-$ϵ$ by selecting the learnt action that maximizes the value function of the matched source task.

Algorithm 2 LSM-based exploration strategy

Input: $targetQ$; $sourceQ$; $ϵ$ Output: Action $$\begin{gathered} Action\leftarrow \left\{\begin{array}{}Argma{x}_a\left[targetQ\right],\mathrm{with}\mathrm{probability} \\ ϵArgma{x}_a\left[sourceQ\right],\mathrm{with}\mathrm{probability}(1-ϵ)\end{array}\right. \end{gathered}$$

This exploratory action advice from LSM-based exploration strategy can reduce the instability from random exploration. Moreover, the modified exploration strategy can greatly speed up the learning of a target task, while achieving at least as good performance as $ϵ$-greedy exploration strategy. The Equation (8) gives a definite proof for this assertion, and the specific proof details of (8) are given in Appendix B.:

$$\begin{array}{cc}\hfill & Q_i^{\pi }(s,a)-Q_i^{{\pi }^{\prime }}(s,a)\le \varrho mi{n}_j\sum _{s^{\prime }}p\sum _{a^{\prime }}\omega \parallel Q_i^{\pi }(s^{\prime },a^{\prime })-Q_j^{\pi }(s^{\prime },a^{\prime })\parallel ,\hfill \end{array}$$

(8)

where p denotes the probability $p\left(s^{\prime }\right|s,a)$ in brief form, w denotes the average probability $(ϵ/\alpha )$ over the total number of actions $\alpha$. $\varrho$ is defined as $\gamma /[1-\gamma (1-ϵ\left)\right]$ with the discount factor $0<\gamma <1$. Q denotes the action value function with its superscript denoting the used exploration strategy, and its subscript denoting the task index. $\pi$ and ${\pi }^{\prime }$ denote respectively the $ϵ$-greedy exploration strategy, and the LSM-based exploration strategy.

4. Experiments

In this section, we conduct several experiments on the maze navigation problem and the mountain car problem to verify the effectiveness of the proposed method. The experiments on maze navigation problem show how knowledge is transferred between similar environments with a tabular representation of discrete state and action space, and action value functions. The experiments on the mountain car problem illustrate a more complex transfer procedure under the continuous control. In this experiment, the radial basis function (RBF) network [36,37], an effective feed-forward neural network with the best approximation and global optimal performance, is utilized to estimate action value functions. Both experiments take Q-Learning framework [38], and the source codes for the experiments can be downloaded from the github link: https://github.com/yizhouzhouyi/LSM.git. The experimental results with the modified $ϵ$-greedy exploration strategy are compared against the results from the traditional $ϵ$-greedy exploration strategy.

4.1. Experiments on Maze Navigation Problem

These experiments are conducted on a 10×10 maze navigation environment in two-dimensional space, as shown in Figure 2. An agent is required to walk from the start point to the target point, while avoiding all obstacles in the path and maximizing the total rewards obtained.

Figure 2. The maze navigation environments of two different settings. The red, yellow and gray grids in the figure denote respectively the starting positions, the target positions, and the obstacles. The goal of the agent is to walk from the start point (the red grids) to the target point (the yellow grids).

The detailed settings about the maze navigation environments are listed below.

• Action: The action space is one-dimensional and discrete. The agent can perform four actions: Upward, downward, leftward, and rightward.
• State: The state space is also one-dimensional and discrete. Since each grid in the maze represents one state, there are a total of 100 states.
• Reward: When the agent reaches the target state, it will receive a reward of +10; when the agent encounters an obstacle, it will get a penalty of −10; and when the agent is in any other state, it will get a living penalty of −0.1.

We design two maze environments as shown in Figure 2. In two maze environments, the action value functions obtained from source tasks (the grids marked in blue in Figure 2a) and the action value functions from target tasks (the grids marked in yellow in Figure 2a,b) are used as the input of LSM.

The experimental results are shown in Figure 3, the learning curves are smoothed with a window of 100 episodes with an error bar at every 100 episodes. The left curve shows the effect of knowledge transfer under the same environment settings, and the right curve shows the effect of knowledge transfer under different environment settings.

Figure 3. The learning curves of LSM-based exploration strategy and $ϵ$-greedy exploration strategy on the maze navigation problem. The curves are smoothed with a window of 100 episodes with an error bar at every 100 episodes.

It can be seen from Figure 3 that LSM-based modified $ϵ$-greedy exploration strategy performs significantly better than normal $ϵ$-greedy exploration strategy. For two exploration strategies, the value of $ϵ$ is set as 0.9.

To have an insight into the effect of $ϵ$, we change the value of $ϵ$ in the range $[0.5,0.9]$ to observe the performance variation of two exploration strategies, and the results are given in Figure 4 and Figure 5. The upper part of Figure 4 shows the average return of 2000 episodes obtained by changing the value of $ϵ$ under the same environment settings, and the lower part of Figure 4 shows the average return of 2000 episodes obtained by changing the value of $ϵ$ under different environment settings. As can be seen, compared with $ϵ$-greedy exploration strategy, the LSM-based exploration strategy can always achieve a higher average return regardless of the value of $ϵ$.

Figure 4. The comparison of the average return of 2000 episodes using the LSM-based and the $ϵ$-greedy exploration strategies, and the values of $ϵ$ are 0.9, 0.7, and 0.5 respectively from left to right. The upper part shows the results under the same environment settings and the lower part shows the results under different environment settings.

Figure 5. The learning curves of LSM-based and $ϵ$-greedy exploration strategies with different values of $ϵ$ ( 0.9, 0.7, and 0.5 from left to right). The curves are smoothed with a window of 100 episodes with an error bar at every 100 episodes.

Figure 5 shows the variation in average return of each episode obtained by changing the value of $ϵ$ under the same environment settings and different environment settings, respectively. When the value of $ϵ$ is set to 0.7 or 0.5, the Q-Learning algorithm using $ϵ$-greedy exploration strategy almost stops learning; however, the Q-Learning algorithm using LSM-based exploration strategy continues to perform well whether under the same environment settings or different environment settings. The reason for this may be that the increase of randomness in action selection greatly will reduce the possibility of taking sequent actions to find valid paths to target state; the good performance of the modified strategy attributes to the knowledge transfer, which can reduce the randomness to some extent.

4.2. Experiments on Mountain Car Problem

These experiments are conducted on the mountain car [39,40] environment under the classical control problem of the OpenAI gym environment library. The mountain car problem belongs to the continuous control problem. As shown in Figure 6, in the experimental environment, a car is located on a two-dimensional track between two mountains; it is required to slide down from the left side of the mountain and rush to the target position on the right side of the mountain.

Figure 6. The mountain car environment. The purpose of the agent is to start from the starting position marked "Start", and use the energy obtained by sliding down from the left side of the mountain to rush to the position marked "Goal" on the right side of the mountain.

The detailed settings about the mountain car environment are listed below:

• Action: The action space is one-dimensional and discrete. The car can perform three actions, including leftward, rightward, and neutral, namely:

  $$Action=(leftward,neutral,rightward).$$
• Reward: Each time the car arrives at a state, it obtains a reward of −1.0.
• State: The state space is two-dimensional and continuous. A state consists of speed and position, defined as:

  $$\left\{\begin{array}{c}Velocity=(-0.007,0.007)\hfill \\ Position=(-1.2,0.6)\hfill \end{array}\right.$$
• Initial state: The position marked "Start" in Figure 6 denotes the initial state of the car. The initial state is set as:

  $$\left\{\begin{array}{c}Position=-0.5\hfill \\ Velocity=0.0\hfill \end{array}\right.$$
• Goal state: The position marked "Goal" in Figure 6 is the target state of the car. The target state is set as: $Position\ge 0.5$.
• State update rule: A state is updated as the rule:

  $$\left\{\begin{array}{c}Velocity=Velocity+0.001\times (Action-1)-0.0025\times cos(3\times Position)\hfill \\ Position=Position+Velocity\hfill \end{array}\right.$$

As state values in these experiments are continuous, we use the RBF network to approximate action value functions instead of the tabular representation. The parameters of estimated action value functions of source and target tasks are the input in LSM, used to find the matched source tasks, from which knowledge is transferred to improve the $ϵ$-greedy exploration strategy.

We compare the traditional $ϵ$-greedy exploration strategy with our strategy, namely the LSM-based exploration strategy, and the experimental results are given in Figure 7. The learning curves are smoothed with a window of five episodes with an error bar at every five episodes. According to the evaluation criteria for knowledge transfer methods in reinforcement learning proposed by Taylor [3], our strategy performs better than the initial performance, called Jumpstart [3], and the final performance, called Asymptotic Performance [3], of the $ϵ$-greedy exploration strategy. In the mountain car environment, because the agent using $ϵ$-greedy exploration strategy explores randomly, it needs to try many times to know which speed it should take to get to the left side of the mountain so that it can use the energy of gliding to reach the right side of the mountain. Further, because of the experience gained in previous tasks, the agent using LSM-based exploration strategy can find a good speed at the beginning, and then uses the feedback from the environment to quickly find the optimal speed. Therefore, our method can achieve better performance at the beginning and the end.

Figure 7. Learning curves for the mountain car problem. The curves are smoothed with a window of five episodes with an error bar at every five episodes.

5. Conclusions

In this paper, we propose a novel task matching algorithm to derive the latent structures of tasks, and align the structures to select highly-matched source tasks. We develop an improved exploration strategy to draw knowledge from the matched source tasks, in order to speed up the learning and to improve the performance of RL agents. We also give a theoretical proof to verify the improvement of the exploration strategy based on latent structure matching. Our method achieves significant performance improvement, when compared with the traditional $ϵ$-greedy exploration strategy on the maze navigation problem and the mountain car problem. Such a method can prove to be useful for real-world applications such as autonomous robots, where sample efficiency is a critical factor. Further, in future work, causal inference may be introduced to transfer knowledge in reinforcement learning by adding causal factors.