Entropic Regularization of Markov Decision Processes

Abstract

An optimal feedback controller for a given Markov decision process (MDP) can in principle be synthesized by value or policy iteration. However, if the system dynamics and the reward function are unknown, a learning agent must discover an optimal controller via direct interaction with the environment. Such interactive data gathering commonly leads to divergence towards dangerous or uninformative regions of the state space unless additional regularization measures are taken. Prior works proposed bounding the information loss measured by the Kullback–Leibler (KL) divergence at every policy improvement step to eliminate instability in the learning dynamics. In this paper, we consider a broader family of f-divergences, and more concretely $\alpha$-divergences, which inherit the beneficial property of providing the policy improvement step in closed form at the same time yielding a corresponding dual objective for policy evaluation. Such entropic proximal policy optimization view gives a unified perspective on compatible actor-critic architectures. In particular, common least-squares value function estimation coupled with advantage-weighted maximum likelihood policy improvement is shown to correspond to the Pearson ${\chi }^2$-divergence penalty. Other actor-critic pairs arise for various choices of the penalty-generating function f. On a concrete instantiation of our framework with the $\alpha$-divergence, we carry out asymptotic analysis of the solutions for different values of $\alpha$ and demonstrate the effects of the divergence function choice on common standard reinforcement learning problems.

1. Introduction

Sequential decision-making problems under uncertainty are described by the mathematical framework of Markov decision processes (MDPs) [1]. The core problem in MDPs is to find an optimal policy—a mapping from states to actions which maximizes the expected cumulative reward collected by an agent over its lifetime. In reinforcement learning (RL), the agent is additionally assumed to have no prior knowledge about the environment dynamics and the reward function [2]. Therefore, direct policy optimization in the RL setting can be seen as a form of stochastic black-box optimization: the agent proposes a query point in the form of a policy, the environment evaluates this point by computing the expected return, after that the agent updates the proposal and the process repeats [3]. There are two conceptual steps in this scheme known as policy evaluation and policy improvement [4]. Both steps require function approximation in high-dimensional and continuous state-action spaces due to the curse of dimensionality [4]. Therefore, statistical learning approaches are employed to approximate the value function of a policy and to perform policy improvement based on the data collected from the environment.

In contrast to traditional supervised learning, in reinforcement learning, the data distribution changes with every policy update. State-of-the-art generalized policy iteration algorithms [5,6,7,8] are mindful of this covariate shift problem [9], taking active measures to account for it. To smoothen the learning dynamics, these algorithms limit the information loss between successive policy updates as measured by the KL divergence or approximations thereof [10]. In the optimization literature, such approaches are categorized as proximal (or trust region) algorithms [11].

The choice of the divergence function determines the geometry of the information manifold [12]. Recently, in particular in the area of implicit generative modeling [13], the choice of the divergence function was shown to have a dramatic effect both on the optimization performance [14] and the perceptual quality of the generated data when various f-divergences were employed [15]. In this paper, we carry over the idea of using generalized entropic proximal mappings [16] given by an f-divergence to reinforcement learning. We show that relative entropy policy search [6], framed as an instance of stochastic mirror descent [17,18] as suggested by [10], can be extended to use any divergence measure from the family of f-divergences. The resulting algorithm provides insights into the compatibility of policy and value function update rules in actor-critic architectures, which we exemplify on several instantiations of the generic f-divergence with representatives from the parametric family of $\alpha$-divergences [19,20,21].

2. Background

This section provides the necessary background on policy gradients [3] and entropic penalties [16] for later derivations and analysis. Standard RL notation [22] is used throughout.

2.1. Policy Gradient Methods

Policy search algorithms [3] commonly use the gradient estimator of the following form [23]

$$\widehat{g}={\widehat{E}}_t\left[{\nabla }_{\theta }log{\pi }_{\theta }{\widehat{A}}_t^w\right]$$

(1)

where ${\pi }_{\theta }\left(a\right|s)$ is a stochastic policy and ${\widehat{A}}_t^w(s_t,a_t)$ is an estimator of the advantage function at timestep t. Expectation ${\widehat{E}}_t[\cdots ]$ indicates an empirical average over a finite batch of samples, in an algorithm that alternates between sampling and optimization. The advantage estimate ${\widehat{A}}_t^w$ in (1) can be obtained from an estimate of the value function [24,25], which in its turn is found by least-squares estimation. Specifically, if $V^w\left(s\right)$ denotes a parametric value function, and if ${\widehat{V}}_t={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k}$ is taken as its rollout-based estimate, then the parameters w can be found as

$$w=arg\underset{\tilde{w}}{min}{\widehat{E}}_t\left[\parallel V^{\tilde{w}}\left(s_t\right)-{\widehat{V}}_t{\parallel }^2\right].$$

(2)

The advantage estimate ${\widehat{A}}_t^w={\sum }_{k=0}^{\infty }{\gamma }^k{\delta }_{t+k}^w$ is then obtained by summing the temporal difference errors ${\delta }_t^w=R_t+\gamma V^w\left(s_{t+1}\right)-V^w\left(s_t\right)$, also known as the Bellman residuals. Treating ${\widehat{A}}_t^w$ as fixed for the purpose of policy improvement, we can view (1) as the gradient of an advantage-weighted log-likelihood; therefore, the policy parameters $\theta$ can be found as

$$\theta =arg\underset{\tilde{\theta }}{max}{\widehat{E}}_t\left[log{\pi }_{\tilde{\theta }}{\widehat{A}}_t^w\right].$$

(3)

Thus, actor-critic algorithms that use the gradient estimator (1) to update the policy can be viewed as instances of the generalized policy iteration scheme, alternating between policy evaluation (2) and policy improvement (3). In the following, we will see that the actor-critic pair (2) and (3), that combines least-squares value function fitting with linear-in-the-advantage-weighted maximum likelihood policy improvement, is just one representative from a family of such actor-critic pairs arising for different choices of the f-divergence penalty within our entropic proximal policy optimization framework.

2.2. Entropic Penalties

The term entropic penalties [16] refers to both f-divergences and Bregman divergences. In this paper, we will focus on f-divergences, leaving generalization to Bregman divergences for future work. The f-divergence [26] between two distributions P and Q with densities p and q is defined as

$$D_f(p\parallel q)=E_q\left[f\left(\frac{p}{q}\right)\right]$$

where f is a convex function on $(0,\infty )$ with $f\left(1\right)=0$ and P is assumed to be absolutely continuous with respect to Q. For example, the KL divergence corresponds to $f_1\left(x\right)=xlogx-(x-1)$, with the formula also applicable to unnormalized distributions [27]. Many common divergences lie on the curve of $\alpha$-divergences [19,20] defined by a special choice of the generator function [21]

$$f_{\alpha }\left(x\right)=\frac{(x^{\alpha }-1)-\alpha (x-1)}{\alpha (\alpha -1)},\alpha \in \mathbb{R}.$$

(4)

The $\alpha$-divergence $D_{\alpha }=D_{f_{\alpha }}$ will be used as the primary example of the f-divergence throughout the paper. For more details on the $\alpha$-divergence and its properties, see Appendix A. Noteworthy is the symmetry of the $\alpha$-divergence with respect to $\alpha =0.5$, which relates reverse divergences as $D_{0.5+\beta }(p\parallel q)=D_{0.5-\beta }(q\parallel p)$.

3. Entropic Proximal Policy Optimization

Consider the average-reward RL setting [2], where the dynamics of an ergodic MDP are given by the transition density $p\left(s^{\prime }\right|s,a)$. An intelligent agent can modulate the system dynamics by sampling actions a from a stochastic policy $\pi \left(a\right|s)$ at every time step of the evolution of the dynamical system. The resulting modulated Markov chain with transition kernel $p_{\pi }\left(s^{\prime }\right|s)={\int }_{A}p\left(s^{\prime }\right|s,a)\pi \left(a\right|s)da$ converges to a stationary state distribution ${\mu }_{\pi }\left(s\right)$ as time goes to infinity. This stationary state distribution induces a state-action distribution ${\rho }_{\pi }(s,a)={\mu }_{\pi }\left(s\right)\pi \left(a\right|s)$, which corresponds to visitation frequencies of state-action pairs [1]. The goal of the agent is to steer the system dynamics to desirable states. Such objective is commonly encoded by the expectation of a random variable $R:S\times A\to \mathbb{R}$ called reward in this context. Thus, the agent seeks a policy that maximizes the expected reward $J\left(\pi \right)=E_{{\rho }_{\pi }(s,a)}\left[R(s,a)\right]$.

In reinforcement learning, neither the reward function R nor the system dynamics $p\left(s^{\prime }\right|s,a)$ are assumed to be known. Therefore, to maximize (or even evaluate) the objective $J\left(\pi \right)$, the agent must sample a batch of experiences in the form of tuples $(s,a,r,s^{\prime })$ from the dynamics and use an empirical estimate $\widehat{J}={\widehat{E}}_t\left[R(s_t,a_t)\right]$ as a surrogate for the original objective. Since the gradient of the expected reward with respect to the policy parameters can be written as [28]

$${\nabla }_{\theta }J\left({\pi }_{\theta }\right)=E_{{\rho }_{{\pi }_{\theta }}(s,a)}[{\nabla }_{\theta }log{\pi }_{\theta }\left(a\right|s)R(s,a)]$$

with a corresponding sample-based counterpart

$${\nabla }_{\theta }\widehat{J}={\widehat{E}}_t[{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t)R(s_t,a_t)],$$

one may be tempted to optimize a sample-based objective

$${\widehat{E}}_t[log{\pi }_{\theta }\left(a_t\right|s_t)R(s_t,a_t)]$$

on a fixed batch of data ${\left\{{(s,a,r,s^{\prime })}_t\right\}}_{t=1}^N$ till convergence. However, such an approach ignores the fact that sampling distribution ${\rho }_{{\pi }_{\theta }}(s,a)$ itself depends on the policy parameters $\theta$; therefore, such greedy optimization aims at a wrong objective [6]. To have the correct objective, the dataset must be sampled anew after every parameter update—doing otherwise will lead to overfitting and divergence. This problem is known in statistics as the covariate shift problem [9].

3.1. Fighting Covariate Shift via Trust Regions

A principled way to account for the change in the sampling distribution at every policy update step is to construct an auxiliary local objective function that can be safely optimized till convergence. Relative entropy policy search (REPS) algorithm [6] proposes a candidate for such an objective

$$J_{\eta }\left(\pi \right)=E_{{\rho }_{\pi }}\left[R\right]-\eta D_1({\rho }_{\pi }\parallel {\rho }_{{\pi }_0})$$

(5)

with ${\pi }_0$ being the current policy under which the data samples were collected, policy $\pi$ being the improvement policy that needs to be found, and $\eta >0$ being a ‘temperature’ parameter that determines how much the next policy can deviate from the current one. The original formulation employs a relative entropy trust region constraint $D_1$ with radius $\epsilon$ instead of a penalty, which allows for finding the optimal temperature $\eta$ as a function of the trust region radius $\epsilon$.

Importantly, the objective function (5) can be optimized in closed form for policy $\pi$ (i.e., treating the policy itself as a variable and not its parameters, in contrast to standard policy gradients). To that end, several constraints on ${\rho }_{\pi }$ are added to ensure stationarity with respect to the given MDP [6]. In a similar vein, we can solve Problem (5) with respect to $\pi$ for any f-divergence with a twice differentiable generator function f.

3.2. Policy Optimization with Entropic Penalties

Following the intuition of REPS, we introduce an f-divergence penalized optimization problem that the learning agent must solve at every policy iteration step

$$\begin{array}{cc}\hfill \underset{\pi }{\mathrm{maximize}}& J_{\eta }\left(\pi \right)=E_{{\rho }_{\pi }}\left[R\right]-\eta D_f({\rho }_{\pi }\parallel {\rho }_{{\pi }_0})\hfill \\ \hfill \mathrm{subject}\mathrm{to}& {\int }_A{\rho }_{\pi }(s^{\prime },a^{\prime })d{a}^{\prime }={\int }_{S\times A}{\rho }_{\pi }(s,a)p\left(s^{\prime }\right|s,a)dsda,\forall s^{\prime }\in S,\hfill \\ & {\int }_{S\times A}{\rho }_{\pi }(s,a)dsda=1,\hfill \\ & {\rho }_{\pi }(s,a)\ge 0,\forall (s,a)\in S\times A.\hfill \end{array}$$

(6)

The agent seeks a policy that maximizes the expected reward and does not deviate from the current policy too much. The first constraint in (6) ensures that the policy is compatible with the system dynamics, and the latter two constraints ensure that $\pi$ is a proper probability distribution. Please note that $\pi$ enters Problem (6) indirectly through ${\rho }_{\pi }$. Since the objective has the form of free energy [29] in ${\rho }_{\pi }$ with an f-divergence playing the role of the usual KL, the solution can be expressed through the derivative of the convex conjugate function $f_{*}^{\prime }$, as shown for general nonlinear problems in [16],

$${\rho }_{\pi }(s,a)={\rho }_{{\pi }_0}(s,a)f_{*}^{\prime }\left(\frac{R(s,a)+{\int }_{S}V\left(s^{\prime }\right)p\left(s^{\prime }\right|s,a)d{s}^{\prime }-V\left(s\right)-\lambda +\kappa (s,a)}{\eta }\right).$$

(7)

Here, $\left\{V\right(s),\lambda ,\kappa (s,a\left)\right\}$ are the Lagrange dual variables corresponding to the three constraints in (6), respectively. Although we get a closed-form solution for ${\rho }_{\pi }$, we still need to solve the dual optimization problem to get the optimal dual variables

$$\begin{array}{cc}\hfill \underset{V,\lambda ,\kappa }{\mathrm{minimize}}& g(V,\lambda ,\kappa )=\eta E_{{\rho }_{{\pi }_0}}\left[f_{*}\left(\frac{A^V(s,a)-\lambda +\kappa (s,a)}{\eta }\right)\right]+\lambda \hfill \\ \hfill \mathrm{subject}\mathrm{to}& \kappa (s,a)\ge 0,\forall (s,a)\in S\times A,\hfill \\ & arg{f}_{*}\in {range}_{x\ge 0}{f}^{\prime }\left(x\right),\forall (s,a)\in S\times A.\hfill \end{array}$$

(8)

Remarkably, the advantage function $A^V(s,a)=R(s,a)+{\int }_{S}V\left(s^{\prime }\right)p\left(s^{\prime }\right|s,a)d{s}^{\prime }-V\left(s\right)$ emerges automatically in the dual objective. The advantage function also appears in the penalty-free linear programming formulation of policy improvement [1], which corresponds to the zero-temperature limit $\eta \to 0$ of our formulation. Thanks to the fact that the dual objective in (8) is given as an expectation with respect to ${\rho }_{{\pi }_0}$, it can be straightforwardly estimated from rollouts. The last constraint in (8) on the argument of $f_{*}$ is easy to evaluate for common $\alpha$-divergences. Indeed, the convex conjugate $f_{\alpha }^{*}$ of the generator function (4) is given by

$$f_{\alpha }^{*}\left(y\right)=\frac{1}{\alpha }{(1+(\alpha -1)y)}^{\frac{\alpha }{\alpha -1}}-\frac{1}{\alpha },\mathrm{for}y(1-\alpha )<1.$$

(9)

Thus, the constraint on $arg{f}_{*}$ in (4) is just a linear inequality $y(1-\alpha )<1$ for any $\alpha$-divergence.

3.3. Value Function Approximation

For small grid-world problems, one can solve Problem (8) exactly for $V\left(s\right)$. However, for larger problems or if the state space is continuous, one must resort to function approximation. Assume we plug an expressive function approximator $V^w\left(s\right)$ in (8), then vector w becomes a new vector of parameters in the dual objective. Later, it will be shown that minimizing the dual when $\eta \to \infty$ is closely related to minimizing the mean squared Bellman error.

3.4. Sample-Based Algorithm for Dual Optimization

To solve Problem (8) in practice, we gather a batch of samples from policy ${\pi }_0$ and replace the expectation in the objective with a sample average. Please note that in principle one also needs to estimate the expectation of the future rewards ${\int }_{S}V\left(s^{\prime }\right)p\left(s^{\prime }\right|s,a)d{s}^{\prime }$. However, since the probability of visiting the same state-action pair in continuous space is zero, one commonly estimates this integral from a single sample [3], which is equivalent to assuming deterministic system dynamics. Inequality constraints in (8) are linear and they must be imposed for every $(s,a)$ pair in the dataset.

3.5. Parametric Policy Fitting

Assume Problem (8) is solved on a current batch of data sampled from ${\pi }_0$ and thus the optimal dual variables $\left\{V\right(s),\lambda ,\kappa (s,a\left)\right\}$ are given. Equation (7) allows one to evaluate the new density ${\rho }_{\pi }(s,a)$ on any pair $(s,a)$ from the dataset. However, it does not yield the new policy $\pi$ directly because representation (7) is variational. A common approach [3] is to assume that the policy is represented by a parameterized conditional density ${\pi }_{\theta }\left(a\right|s)$ and fit this density to the data using maximum likelihood.

To fit a parametric density ${\pi }_{\theta }\left(a\right|s)$ to the true solution $\pi \left(a\right|s)$ given by (7), we minimize the KL divergence $D_1({\rho }_{\pi }\parallel {\rho }_{{\pi }_{\theta }})$. Minimization of this KL is equivalent to maximization of the weighted maximum likelihood $\widehat{E}[f_{*}^{\prime }(\cdots )log{\rho }_{{\pi }_{\theta }}]$. Unfortunately, distribution ${\rho }_{{\pi }_{\theta }}(s,a)={\mu }_{{\pi }_{\theta }}\left(s\right){\pi }_{\theta }\left(a\right|s)$ is in general not known because ${\mu }_{{\pi }_{\theta }}\left(s\right)$ does not only depend on the policy but also on the system dynamics. Assuming the effect of policy parameters on the stationary state distribution is small [3], we arrive at the following optimization problem for fitting the policy parameters

$$\theta =arg\underset{\tilde{\theta }}{max}{\widehat{E}}_t\left[log{\pi }_{\tilde{\theta }}\left(a_t\right|s_t)f_{*}^{\prime }\left(\frac{{\widehat{A}}^w(s_t,a_t)-\lambda +\kappa (s_t,a_t)}{\eta }\right)\right].$$

(10)

Compare our policy improvement step (10) to the commonly used advantage-weighted maximum likelihood (ML) objective (3). They look surprisingly similar (especially if $f_{*}^{\prime }\left(y\right)=y$ is a linear function), which is not a coincidence and will be systematically explained in the next sections.

3.6. Temperature Scheduling

The ‘temperature’ parameter $\eta$ trades off reward vs divergence, as can be seen in the objective function in Problem (6). In practice, devising a schedule for $\eta$ may be hard because $\eta$ is sensitive to reward scaling and policy parameterization. A more intuitive way to impose the f-divergence proximity condition is by adding it as a constraint $D_f({\rho }_{\pi }\parallel {\rho }_{{\pi }_0})\le \epsilon$ with a fixed $\epsilon$ and then treating the temperature $\eta \ge 0$ as an optimization variable. Such formulation is easy to incorporate into the dual (8) by adding a term $\eta \epsilon$ to the objective and a constraint $\eta \ge 0$ to the list of constraints. Constraint-based formulation was successfully used before with a KL divergence constraint [6] and with its quadratic approximation [5,7].

3.7. Practical Algorithm for Continuous State-Action Spaces

Our proposed approach for entropic proximal policy optimization is summarized in Algorithm 1. Following the generalized policy iteration scheme, we (i) collect data under a given policy, (ii) evaluate the policy by solving (8), and (iii) improve the policy by solving (10). In the following section, several instantiations of Algorithm 1 with different choices of function f will be presented and studied.

Algorithm 1: Primal-dual entropic proximal policy optimization with function approximation

4. High- and Low-Temperature Limits; $\alpha$-Divergences; Analytic Solutions and Asymptotics

How does the f-divergence penalty influence policy optimization? How should one choose the generator function f? What role does the step size play in optimization? This section will try to answer these and related questions. First, two special choices of the penalty function f are presented, which reveal that the common practice of using mean squared Bellman error minimization coupled with advantage reweighted policy update is equivalent to imposing a Pearson ${\chi }^2$-divergence penalty. Second, high- and low-temperature limits are studied, on one hand revealing the special role the Pearson ${\chi }^2$-divergence plays, being the high-temperature limit of all smooth f-divergences, and on the other hand establishing a link to the linear programming formulation of policy search as the low-temperature limit of our entropic penalty-based framework.

4.1. KL Divergence ($\alpha =1$) and Pearson ${\chi }^2$-Divergence ($\alpha =2$)

As can be deduced from the form of (10), great simplifications occur when $f_{*}^{\prime }\left(y\right)$ is a linear function ($\alpha =2$, see (9)) or an exponential function ($\alpha =1$). The fundamental reason for such simplifications lies in the fact that linear and exponential functions are homomorphisms with respect to addition. This allows, in particular, discovery of a closed-form solution for the dual variable $\lambda$ and thus eliminate it from the optimization. Moreover, in these two special cases, the dual variables $\kappa (s,a)$ can also be eliminated. They are responsible for non-negativity of probabilities: when $\alpha =1$ (KL), $\kappa (s,a)=0$ uniformly for all $\eta \ge 0$, when $\alpha =2$ (Pearson), $\kappa (s,a)=0$ for sufficiently big $\eta$.

Table 1. Empirical policy evaluation and policy improvement objectives for $\alpha \in \{1,2\}$.

KL Divergence ($\mathit{\alpha }=1$)	Pearson ${\mathit{\chi }}^2$-Divergence ($\mathit{\alpha }=2$)
${\widehat{g}}_1\left(w\right)=\eta log\left({\widehat{E}}_t\left[exp\left(\frac{{\widehat{A}}^w(s_t,a_t)}{\eta }\right)\right]\right)$	${\widehat{g}}_2\left(w\right)=\frac{1}{2\eta }{\widehat{E}}_t\left[{\left({\widehat{A}}^w(s_t,a_t)-{\widehat{E}}_t\left[{\widehat{A}}^w\right]\right)}^2\right]$
${\widehat{L}}_1\left(\theta \right)={\widehat{E}}_t\left[log{\pi }_{\theta }\left(a_t\right|s_t)exp\left(\frac{{\widehat{A}}^w(s_t,a_t)-{\widehat{g}}_1\left(w\right)}{\eta }\right)\right]$	${\widehat{L}}_2\left(\theta \right)=\frac{1}{\eta }{\widehat{E}}_t\left[log{\pi }_{\theta }\left(a_t\right|s_t)\left({\widehat{A}}^w(s_t,a_t)-{\widehat{E}}_t\left[{\widehat{A}}^w\right]+\eta \right)\right]$

gives the corresponding empirical actor-critic optimization objective pairs. A generic primal-dual actor-critic algorithm with an $\alpha$-divergence penalty performs two steps

$$\begin{array}{cccc}\left(\mathrm{step}1:\mathrm{policy}\mathrm{evaluation}\right)\hfill & \hfill & \hfill \underset{w}{\mathrm{minimize}}& {\widehat{g}}_{\alpha }\left(w\right)\hfill \\ \left(\mathrm{step}2:\mathrm{policy}\mathrm{improvement}\right)\hfill & \hfill & \hfill \underset{\theta }{\mathrm{maximize}}& {\widehat{L}}_{\alpha }\left(\theta \right)\hfill \end{array}$$

inside a policy iteration loop. It is worth comparing the explicit formulas in Table 1 to the customarily used objectives (2) and (3). To make the comparison fair, notice that (2) and (3) correspond to discounted infinite horizon formulation with discount factor $\gamma \in (0,1)$ whereas formulas in Table 1 are derived for the average-reward setting. In general, the difference between these two settings can be ascribed to an additional baseline that must be subtracted in the average reward setting [2]. In our derivations, the baseline corresponds to the dual variable $\lambda$, as in classical linear programming formulation of policy iteration [1], and it is automatically gets subtracted from the advantage (see (8)).

Mean Squared Error Minimization with Advantage Reweighting is Equivalent to Pearson Penalty

The baseline for $\alpha =2$ is given by the average advantage ${\lambda }_2={\widehat{E}}_t\left[{\widehat{A}}^w(s_t,a_t)\right]$, which also equals the average return in our setting [1,2]. Therefore, to translate the formulas from Table 1 to the discounted infinite horizon form (2) and (3), we need to remove the baseline and add discounting to the advantage; that is, set $A^w(s,a)=R(s,a)+\gamma {\int }_S{V}^w\left(s^{\prime }\right)p\left(s^{\prime }\right|s,a)d{s}^{\prime }-V^w\left(s\right)$. Then the dual objective

$${\widehat{g}}_2\left(w\right)\propto {\widehat{E}}_t\left[{\left({\widehat{A}}^w(s_t,a_t)\right)}^2\right]$$

(11)

is proportional to the average squared advantage. Naive optimization of (11) leads to the family of residual gradient algorithms [30,31]. However, if the same Monte Carlo estimate of the value function is used as in (2), then (11) and (2) are exactly equivalent. The same holds for the Pearson actor

$${\widehat{L}}_2\left(\theta \right)\propto {\widehat{E}}_t\left[log{\pi }_{\theta }\left(a_t\right|s_t){\widehat{A}}^w(s_t,a_t)\right]$$

(12)

and the standard policy improvement (3) provided that $\eta ={\widehat{E}}_t\left[{\widehat{A}}^w(s_t,a_t)\right]$. That means (12) is equivalent to (3) if the weight of the divergence penalty is equal to the expected return.

4.2. High- and Low-Temperature Limits

In the previous subsection, we established a direct correspondence between the least-squares value function fitting coupled with the advantage-weighted maximum likelihood policy parameters estimation (2) and (3) and the dual-primal pair of optimization problems (11) and (12) arising from our Algorithm 1 for the special choice of the Pearson ${\chi }^2$-divergence penalty. In this subsection, we will show that this is not a coincidence but a manifestation of the fundamental fact that the Pearson ${\chi }^2$-divergence is the quadratic approximation of any smooth f-divergence about unity.

4.2.1. High Temperatures: All Smooth f-Divergences Tend Towards Pearson ${\chi }^2$-Divergence

There are two ways to show the independence of the primal-dual solution (8)–(10) on the choice of the divergence penalty: either exactly solve an approximate problem or approximate the exact solution of the original problem. In the first case, the penalty is replaced with its Taylor expansion at $\eta \to \infty$, which turns out to be the Pearson ${\chi }^2$-divergence, and then the derivation becomes equivalent to the natural policy gradient derivation [5]. In the second case, the exact solution (8)–(10) is expanded by Taylor: for big $\eta$, dual variables $\kappa (s,a)$ can be dropped if ${\rho }_{{\pi }_0}(s,a)>0$, which yields

$$f_{*}\left(\frac{A^w(s,a)-\lambda }{\eta }\right)=f_{*}\left(0\right)+\frac{A^w(s,a)-\lambda }{\eta }{f}_{*}^{\prime }\left(0\right)+\frac{1}{2}{\left(\frac{A^w(s,a)-\lambda }{\eta }\right)}^2{f}_{*}^{″}\left(0\right)+o\left(\frac{1}{{\eta }^2}\right).$$

(13)

By definition of the f-divergence, the generator function f satisfies the condition $f\left(1\right)=0$. Without loss of generality [32], one can impose an additional constraint $f^{\prime }\left(1\right)=0$ for convenience. Such constraint ensures that the graph of the function $f\left(x\right)$ lies entirely in the upper half-plane, touching the x-axis at a single point $x=1$. From the definition of the convex conjugate $f_{*}^{\prime }={\left(f^{\prime }\right)}^{-1}$, we can deduce that $f_{*}^{\prime }\left(0\right)=1$ and $f_{*}\left(0\right)=0$; by rescaling, it is moreover possible to set $f^{″}\left(1\right)=f_{*}^{″}\left(0\right)=1$. These properties are automatically satisfied by the $\alpha$-divergence, which can be verified by a direct computation. With this in mind, it is straightforward to see that substitution of (13) into (8) yields precisely the quadratic objective ${\widehat{g}}_2\left(w\right)$ from Table 1, the difference being of the second order in $1/\eta$.

To obtain the asymptotic policy update objective, one can expand (10) in the high-temperature limit $\eta \to \infty$ and observe that it equals ${\widehat{L}}_2\left(\theta \right)$ from Table 1 with the difference being of the second order in $1/\eta$. Therefore, it is established that the choice of the divergence function plays a minor role for big temperatures (small policy update steps). Since this is the mode in which the majority of iterative algorithms operate, our entropic proximal policy optimization point of view provides a rigorous justification for the common practice of using the mean squared Bellman error objective for value function fitting and the advantage-weighted maximum likelihood objective for policy improvement.

4.2.2. Low Temperatures: Linear Programming Formulation Emerges in the Limit

Setting $\eta$ to a small number is equivalent to allowing large policy update steps because $\eta$ is the weight of the divergence penalty in the objective function (6). Such regime is rather undesirable in reinforcement learning because of the covariate shift problem mentioned in the introduction. Problem (6) for $\eta \to 0$ turns into a well-studied linear programming formulation [1,10] that can be readily applied if the model $\{p\left(s^{\prime }\right|s,a),R(s,a)\}$ is known.

It is not straightforward to derive the asymptotics of policy evaluation (8) and policy improvement (10) for a general smooth f-divergence in the low-temperature limit $\eta \to 0$ because the dual variables $\kappa (s,a)$ do not disappear, in contrast to the high-temperature limit (13). However, for the KL divergence penalty (see Table 1), one can show that the policy evaluation objective $g_1\left(w\right)$ tends towards the supremum of the advantage $g_1\left(w\right)\to {sup}_{s,a}{A}^w(s,a)$; the optimal policy is deterministic, $\pi \left(a\right|s)\to \delta (a-arg{sup}_b{A}^w(s,b))$, therefore $L\left(\theta \right)\to log{\pi }_{\theta }\left(\overline{a}\right|\overline{s})$ with $(\overline{s},\overline{a})=arg{sup}_{s^{\prime },a^{\prime }}{A}^w(s^{\prime },a^{\prime })$.

5. Empirical Evaluations

To develop an intuition regarding the influence of the entropic penalties on policy improvement, we first consider a simplified version of the reinforcement learning problem—namely the stochastic multi-armed bandit problem [33]. In this setting, our algorithm is closely related to the family of Exp3 algorithms [34], originally motivated by the adversarial bandit problem. Subsequently, we evaluate our approach in the standard reinforcement learning setting.

5.1. Illustrative Experiments on Stochastic Multi-Armed Bandit Problems

In the stochastic multi-armed bandit problem [33], at every time step $t\in \{1,\cdots ,T\}$, an agent chooses among K actions $a\in \mathcal{A}$. After every choice $a_t=a$, it receives a noisy reward $R_t=R\left(a_t\right)$ drawn from a distribution with mean $Q\left(a\right)$. The goal of the agent is to maximize the expected total reward $J=E\left[{\sum }_{t=1}^T{R}_t\right]$. Given the true values $Q\left(a\right)$, the optimal strategy is to always choose the best action, $a_t^{*}={arg\; max}_{a}Q\left(a\right)$. However, due to the lack of knowledge, the agent faces the exploration-exploitation dilemma. A generic way to encode the exploration-exploitation trade-off is by introducing a policy ${\pi }_t$, i.e., a distribution from which the agent draws actions $a_t\sim {\pi }_t$. Thus, the question becomes: given the current policy ${\pi }_t$ and the current estimate of action values ${\widehat{Q}}_t$, what should the policy ${\pi }_{t+1}$ at the next time step be? Unlike the choice of the best action under perfect information, such sampling policies are hard to derive from first principles [35].

We apply our generic Algorithm 1 to the stochastic multi-armed bandit problem to illustrate the effects of the divergence choice. The value function disappears because there is no state and no system dynamics in this problem. Therefore, the estimate ${\widehat{Q}}_t$ plays the role of the advantage, and the dual optimization (8) is performed only with respect to the remaining Lagrange multipliers.

5.1.1. Effects of $\alpha$ on Policy Improvement

Figure 1 shows the effects of the $\alpha$-divergence choice on policy updates. We consider a 10-armed bandit problem with arm values $Q\left(a\right)\sim \mathcal{N}(0,1)$ and keep the temperature fixed at $\eta =2$ for all values of $\alpha$. Several iterations starting from an initial uniform policy are shown in the figure for comparison. Extremely large positive and negative values of $\alpha$ result in $\epsilon$-elimination and $\epsilon$-greedy policies, respectively. Small values of $\alpha$, in contrast, weigh actions according to their values. Policies for $\alpha <1$ are peaked and heavy-tailed, eventually turning into $\epsilon$-greedy policies when $\alpha \to -\infty$. Policies for $\alpha \ge 1$ are more uniform, but they put zero mass on bad actions, eventually turning into $\epsilon$-elimination policies when $\alpha \to \infty$. For $\alpha \ge 1$, policy iteration may spend a lot of time in the end deciding between two best actions, whereas for $\alpha <1$ the final convergence is faster.

5.1.2. Effects of $\alpha$ on Regret

The average regret $C_n=n{Q}_{max}-\mathbb{E}\left[{\sum }_{t=0}^{n-1}{R}_t\right]$ is shown in Figure 2 for different values of $\alpha$ as a function of the time step n with $95\%$ confidence error bars. The performance of the UCB algorithm [33] is also shown for comparison. The presented results are obtained in a 20-armed bandit environment where rewards have Gaussian distribution $R\left(a\right)\sim \mathcal{N}\left(Q\right(a),0.5)$. Arm values are estimated from observed rewards and the policy is updated every 20 time steps. The temperature parameter $\eta$ is decreased starting from $\eta =1$ after every policy update according to the schedule ${\eta }^{+}=\beta \eta$ with $\beta =0.8$. Results are averaged over 400 runs. In general, extreme $\alpha$’s accumulate more regret. However, they eventually focus on a single action and flatten out. Small $\alpha$’s accumulate less regret, but they may keep exploring sub-optimal actions longer. Values of $\alpha \in [0,2]$ perform comparably with UCB after around 400 steps, once reliable estimates of values have been obtained.

Figure 3 shows the average regret after a given number of time steps as a function of the divergence type $\alpha$. As can be seen from the figure, smaller values of $\alpha$ result in lower regret. Large negative $\alpha$’s correspond to $\epsilon$-greedy policies, which oftentimes prematurely converge to a sub-optimal action, failing to discover the optimal action for a long time if the exploration probability $\epsilon$ is small. Large positive $\alpha$’s correspond to $\epsilon$-elimination policies, which may by mistake completely eliminate the best action or spend a lot of time deciding between two options in the end of learning, accumulating more regret. The optimal value of the parameter $\alpha$ depends on the time horizon for which the policy is being optimized. Depending on the horizon, the minimum of the curves shifts from slightly negative $\alpha$’s towards the range $\alpha \in [0,2]$ with increasing time horizon.

5.2. Empirical Evaluations on Ergodic MDPs

We evaluate our policy iteration algorithm with f-divergence on standard grid-world reinforcement learning problems from OpenAI Gym [36]. The environments that terminate or have absorbing states are restarted during data collection to ensure ergodicity. Figure 4 demonstrates the learning dynamics on different environments for various choices of the divergence function. Parameter settings and other implementation details can be found in Appendix B. In summary, one can either promote risk averse behavior by choosing $\alpha <0$, which may, however, result in sub-optimal exploration, or one can promote risk seeking behavior with $\alpha >1$, which may lead to overly aggressive elimination of options. Our experiments suggest that the optimal balance should be found in the range $\alpha \in [0,1]$. It should be noted that the effect of the $\alpha$-divergence on policy iteration is not linear and not symmetric with respect to $\alpha =0.5$, contrary to what one could have expected given the symmetry of the $\alpha$-divergence as a function of $\alpha$. For example, switching from $\alpha =-3$ to $\alpha =-2$ may have little effect on policy iteration, whereas switching from $\alpha =3$ to $\alpha =4$ may have a much more pronounced influence on the learning dynamics.

6. Related Work

Apart from computational advantages, information-theoretic approaches provide a solid framework for describing and studying aspects of intelligent behavior [37], from autonomy [38] and curiosity [39] to bounded rationality [40] and game theory [41].

Entropic proximal mappings were introduced in [16] as a general framework for constructing approximation and smoothing schemes for optimization problem. Problem formulation (6) presented here can be considered as an application of this general theory to policy optimization in Markov decision processes. Following the recent work [10] that establishes links between popular in reinforcement learning KL-divergence-regularized policy iteration algorithms [6,7] and a well-known in optimization stochastic mirror descent algorithm [17,18], one can view our Algorithm 1 as an analog of the mirror descent with an f-divergence penalty.

Concurrent works [42,43] consider similar regularized formulations, although in the policy space instead of the state-action distribution space and in the infinite horizon discounted setting instead of the average-reward setting. The $\alpha$-divergence in its entropic form, i.e., when the base measure is a uniform distribution, was used in several papers under the name Tsallis entropy [44,45,46,47], where its sparsifying effect was exploited in large discrete action spaces.

An alternative proximal reinforcement learning scheme was introduced in [48] based on the extragradient method for solving variational inequalities and leveraging operator splitting techniques. Although the idea of exploiting proximal maps and updates in the primal and dual spaces is similar to ours, regularization in [48] is applied in the value function space to smoothen generalized TD learning algorithms, whereas we study regularization in the primal space.

7. Conclusions

We presented a framework for deriving actor-critic algorithms as pairs of primal-dual optimization problems resulting from regularization of the standard expected return objective with so-called entropic penalties in the form of an f-divergence. Several examples with $\alpha$-divergence penalties have been worked out in detail. In the limit of small policy update steps, all f-divergences with twice differentiable generator function f are approximated by the Pearson ${\chi }^2$-divergence, which was shown to yield the most commonly used in reinforcement learning pair of actor-critic updates. Thus, our framework provides a sound justification for the common practice of minimizing mean squared Bellman error in the policy evaluation step and fitting policy parameters by advantage-weighted maximum likelihood in the policy improvement step.

In the future work, incorporating non-differentiable generator functions, such as the absolute value that corresponds to the total variation distance, may provide a principled explanation for the empirical success of the algorithms not accounted for by our current smooth f-divergence framework, such as the proximal policy optimization algorithm [8]. Establishing a tighter connection between online convex optimization that employs Bregman divergences and reinforcement learning will likely yield both a deeper understanding of the optimization dynamics in RL and allow for improved practical algorithms building on the firm fundament of optimization theory.