Risk-Sensitive Policy with Distributional Reinforcement Learning

Abstract

Classical reinforcement learning (RL) techniques are generally concerned with the design of decision-making policies driven by the maximisation of the expected outcome. Nevertheless, this approach does not take into consideration the potential risk associated with the actions taken, which may be critical in certain applications. To address that issue, the present research work introduces a novel methodology based on distributional RL to derive sequential decision-making policies that are sensitive to the risk, the latter being modelled by the tail of the return probability distribution. The core idea is to replace the Q function generally standing at the core of learning schemes in RL by another function, taking into account both the expected return and the risk. Named the risk-based utility function U, it can be extracted from the random return distribution Z naturally learnt by any distributional RL algorithm. This enables the spanning of the complete potential trade-off between risk minimisation and expected return maximisation, in contrast to fully risk-averse methodologies. Fundamentally, this research yields a truly practical and accessible solution for learning risk-sensitive policies with minimal modification to the distributional RL algorithm, with an emphasis on the interpretability of the resulting decision-making process.

1. Introduction

The reinforcement learning (RL) approach is concerned with the learning process of a sequential decision-making policy based on the interactions between an agent and its environment [1]. More precisely, the training is based on the rewards acquired by the agent from the environment as a consequence of its actions. Within this context, the objective is to identify the actions maximising the discounted sum of rewards, also named return. Multiple sound approaches exist, based on the RL paradigm, and key successes/milestones have been achieved throughout the years of research. Nevertheless, these RL algorithms mostly rely on the expectation of the return, not on its complete probability distribution. For instance, the popular Q-learning methodology is based on the modelling of the Q function, which can be seen as an estimation of the expected return [2].

While focusing exclusively on the expectation of the return has already proven to be perfectly sound for numerous applications, this approach exhibits clear limitations for other decision-making problems. Indeed, some areas of application may also require properly mitigating the risk associated with the actions taken [3]. One could, for instance, mention the healthcare [4] and finance [5] sectors, but also robotics in general [6], and especially autonomous driving [7]. For the sake of clarity, two concrete examples are provided as illustrations. In finance, during a trading activity in a market, the objective is not to solely maximise profit but also to properly monitor and mitigate the risk. In robotics, when navigating in an environment, a robot has to ensure that its decision-making will not induce an excessive level of risk, which could potentially harm its integrity. This concern holds true in diverse environments, ranging from the harsh and inhospitable conditions of planet Mars to everyday scenarios like autonomous driving on roads. In fact, there a myriad of sequential decision-making problems exist, which require an appropriate evaluation and mitigation of the risk. Furthermore, such a requirement for risk management may not only be true for the decision-making policy, but potentially also for the exploration policy during the learning process. In addition, properly taking risk into consideration may be particularly convenient in environments characterised by substantial uncertainty.

This research work suggests taking advantage of the distributional RL approach, belonging to the Q-learning category, in order to learn risk-sensitive policies. Basically, a distributional RL algorithm targets the complete probability distribution of the random return rather than only its expectation [8]. This methodology presents key advantages. Firstly, it enables the learning of a richer representation of the environment, which may lead to an increase in the decision-making policy performance. Secondly, the distributional RL approach contributes to improving the explainability of the decision-making process, which is key in machine learning to avoid black-box models. Lastly and most importantly for this research work, it makes possible the convenient derivation of decision-making policies, as well as exploration strategies that are sensitive to the risk.

The core idea promoted by this research work is the use of the risk-based utility function U as a replacement for the popular Q function for action selection. In fact, it may be seen as an extension of the Q function that considers the risk, which is assumed to be represented by the worst returns that are achievable by a policy. Therefore, the function U is to be derived from the complete probability distribution of the random return Z, which is learnt by any distributional RL algorithm. The single modification to that RL algorithm to learn risk-sensitive policies is to employ the utility function U rather than the expected return Q for both exploration and decision-making. This enables the presented approach to become a very practical and interpretable RL solution for risk-sensitive decision making.

2. Literature Review

The core objective of the classical RL approach is to learn optimal decision-making policies without any concerns about the risk or safety [1]. The resulting policies are said to be risk-neutral. Nevertheless, there are numerous real-world applications requiring to take the risk into consideration in order to ensure safer decision making [3]. Two main approaches can be identified to achieving a safe RL. Firstly, the optimality criterion can be modified so that a safety factor is included. Secondly, the exploration process can be altered based on a risk metric [9]. These techniques give rise to risk-sensitive or risk-averse policies.

Scientific research on risk-sensitive RL has been particularly active for the past decade. Various relevant risk criteria have been studied for that purpose. The most popular ones are undoubtedly the mean-variance [10,11,12] and the (Conditional) Value at Risk (CVaR) [13,14,15]. Innovative techniques have been introduced for both policy gradient [16,17,18] and value iteration [19,20,21,22,23] approaches, with the proposed solutions covering both discrete and continuous action spaces. Additionally, risk-sensitive methodologies have also been studied in certain niche sub-fields of RL, such as robust adversarial RL [24], but also multi-agent RL [25].

Focusing on the value iteration methodology, the novel distributional RL approach [8] has been a key breakthrough, by providing access to the full probability distribution of the random return. For instance, [20] suggests achieving risk-sensitive decision-making via a distortion risk measure. Applied on top of the IQN distributional RL algorithm, this is, in fact, equivalent to changing the sampling distribution of the quantiles. In [21], a novel actor-critic framework is presented, based on the distributional RL approach for the critic component. The latter work is extended to the offline setting in [22], since training RL agents online may be prohibitive because of the risk inevitably induced by exploration. Additionally, [23] introduces the Worst-Case Soft Actor Critic (WCSAC) algorithm, which is based on the approximation of the probability distribution of accumulated safety-costs in order to achieve risk control. More precisely, a certain level of CVaR, estimated from the distribution, is regarded as a safety constraint.

In light of this literature, the novel solution introduced in this research paper presents key advantages. Firstly, the methodology proposed is relatively simple and can be applied on top of any distributional RL algorithm with minimal modifications to the core algorithm. Secondly, the proposed approach can span the entire potential trade-off between risk minimisation and expected return maximisation. According to the user’s needs, the policy learnt can be risk-averse, risk-neutral or in between the two (risk-sensitive). Lastly, the solution presented improves the interpretability of the decision-making policy that was learnt.

3. Materials and Methods

3.1. Theoretical Background

3.1.1. Markov Decision Process

In RL, the interactions between the agent and its environment are generally modelled as a Markov decision process (MDP). An MDP is a 6-tuple $(\mathcal{S},\mathcal{A},p_R,p_T,p_0,\gamma )$ where $\mathcal{S}$ and $\mathcal{A}$, respectively, are the state and action spaces, $p_R\left(r\right|s,a)$ is the probability distribution from which the reward $r\in \mathbb{R}$ is drawn given a state–action pair $(s,a)$, $p_T\left(s^{\prime }\right|s,a)$ is the transition probability distribution, $p_0\left(s_0\right)$ is the probability distribution over the initial states $s_0\in \mathcal{S}$, and $\gamma \in [0,1]$ is the discount factor. The RL agent makes decisions according to its policy $\pi :\mathcal{S}\to \mathcal{A}$, which is assumed to be deterministic, mapping the states $s\in \mathcal{S}$ to the actions $a\in \mathcal{A}$.

3.1.2. Distributional Reinforcement Learning

In classical Q-learning RL, the core idea is to model the state–action value function $Q^{\pi }:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ of a policy $\pi$. This important quantity $Q^{\pi }(s,a)$ represents the expected discounted sum of rewards obtained by executing an action $a\in \mathcal{A}$ in a state $s\in \mathcal{S}$ and then following a policy $\pi$:

$$Q^{\pi }(s,a)=\underset{s_t,r_t}{\mathbb{E}}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right],(s_0,a_0):=(s,a),a_t=\pi \left(s_t\right).$$

(1)

The key to the learning process is the Bellman equation [26] that the Q function satisfies:

$$Q^{\pi }(s,a)=\underset{s^{\prime },r}{\mathbb{E}}\left[r+\gamma Q^{\pi }(s^{\prime },\pi \left(s^{\prime }\right))\right].$$

(2)

In classical RL, the main objective is to determine an optimal policy ${\pi }^{*}$, which can be defined based on the optimal state–action value function $Q^{*}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$, as follows:

$$Q^{*}(s,a)=\underset{s^{\prime },r}{\mathbb{E}}\left[r+\gamma \underset{a^{\prime }\in \mathcal{A}}{max}{Q}^{*}(s^{\prime },a^{\prime })\right],$$

(3)

$${\pi }^{*}\left(s\right)\in \underset{a\in \mathcal{A}}{argmax}{Q}^{*}(s,a).$$

(4)

The optimal policy ${\pi }^{*}$ maximises the expected return (discounted sum of rewards). This research work later presents an alternative objective criterion for optimality in a risk-sensitive RL setting.

The distributional RL approach goes a step further by modelling the full probability distribution over returns instead of only its expectation [8], as illustrated in Figure 1. To this end, let the reward $R(s,a)$ be a random variable distributed under $p_R(·|s,a)$, the state-action value distribution $Z^{\pi }\in \mathcal{Z}$ of a policy $\pi$ (also called state-action return distribution function) is a random variable, defined as follows:

$$Z^{\pi }(s,a)\stackrel{D}{=}\sum _{t=0}^{\infty }{\gamma }^{t}R(s_t,a_t),(s_0,a_0):=(s,a),a_t=\pi \left(s_t\right),s_{t+1}\sim p_T(·|s_t,a_t),$$

(5)

where $A\stackrel{D}{=}B$ denotes the equality in probability distribution between the random variables A and B. Therefore, the state–action value function $Q^{\pi }$ is the expectation of the random return $Z^{\pi }$. Equivalent to the expected case, there exists a distributional Bellman equation that recursively describes the random return $Z^{\pi }$ of interest:

$$Z^{\pi }(s,a)\stackrel{D}{=}R(s,a)+\gamma P^{\pi }{Z}^{\pi }(s,a),$$

(6)

$$P^{\pi }{Z}^{\pi }(s,a):\stackrel{D}{=}{Z}^{\pi }(s^{\prime },a^{\prime }),s^{\prime }\sim p_T(·|s,a),a^{\prime }=\pi \left(s^{\prime }\right),$$

(7)

where $P^{\pi }:\mathcal{Z}\to \mathcal{Z}$ is the transition operator. To end this section, one can define the distributional Bellman operator ${\mathcal{T}}^{\pi }:\mathcal{Z}\to \mathcal{Z}$, together with the distributional Bellman optimality operator ${\mathcal{T}}^{*}:\mathcal{Z}\to \mathcal{Z}$ as follows:

$${\mathcal{T}}^{\pi }{Z}^{\pi }(s,a)\stackrel{D}{=}R(s,a)+\gamma P^{\pi }{Z}^{\pi }(s,a),$$

(8)

$${\mathcal{T}}^{*}{Z}^{*}(s,a)\stackrel{D}{=}R(s,a)+\gamma Z^{*}\left(s^{\prime },{\pi }^{*}\left(s^{\prime }\right)\right),s^{\prime }\sim p_T(·|s,a).$$

(9)

A distributional RL algorithm may be characterised by two core features. Firstly, both the representation and parameterisation of the random return probability distribution have to be properly selected. Multiple solutions exist representing a unidimensional distribution: probability density function (PDF), cumulative distribution function (CDF), quantile function (QF). In practice, deep neural networks (DNNs) are generally used for the approximation of these particular functions. The second key feature relates to the probability metric adopted for comparing two distributions, such as the Kullback–Leibler (KL) divergence, the Cramer distance or the Wasserstein distance. More precisely, the role of the probability metric in distributional RL is to quantitatively compare two probability distributions of the random return so that a temporal difference (TD) learning method is applied in a similar way to the mean squared error between Q-values in classical RL. The probability metric plays an even more important role, as different metrics offer distinct theoretical convergence guarantees for distributional RLs.

3.2. Methodology

3.2.1. Objective Criterion for Risk-Sensitive RL

As previously explained, the objective in classical RL is to learn a decision-making policy $\pi \in \mathsf{\Pi }$ that maximises the expectation regarding the discounted sum of rewards. Formally, this objective criterion can be expressed as the following:

$$\underset{\pi }{\mathrm{maximise}}\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right].$$

(10)

In order to effectively take into consideration the risk and value its mitigation, this research work presents an update for the former objective. In fact, coming up with a generic definition for the risk is not trivial since the risk is generally dependent on the decision-making problem itself. In the present research work, it is assumed that the risk is assessed on the basis of the worst returns achievable by a policy $\pi$. Therefore, a successful decision-making policy should ideally maximise the expected discounted sum of rewards while avoiding low values for the worst-case returns. The latter requirement is approximated with a new constraint attached to the former objective, defined in Equation (10): the probability of having the policy achieving a lower return than a certain minimum value should not exceed a given threshold. Mathematically, the alternative objective criterion proposed for risk-sensitive RL can be expressed as follows:

$$\begin{array}{cccc}\hfill & \underset{\pi }{\mathrm{maximise}}\hfill & \hfill & \mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right],\hfill \\ \hfill & \mathrm{such}\mathrm{that}\hfill & \hfill & p\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\le R_{\min }\right]\le ϵ,\hfill \end{array}$$

(11)

where:

•

$p[\star ]$ denotes the probability of the event ⋆,

•

$R_{\min }$ is the minimum acceptable return (from the perspective of risk mitigation),

•

$ϵ\in [0,1]$ is the threshold probability that is not to be exceeded.

3.2.2. Practical Modelling of the Risk

As previously hinted, this research work assumes that the risk associated with an action is related to the worst achievable returns when executing that particular action and then following a certain decision-making policy $\pi$. In such a context, the distributional RL approach becomes particularly interesting, by providing access to the full probability distribution of the random return $Z^{\pi }$. Thus, the risk can be efficiently assessed by examining the so-called tail of the learnt probability distribution. Moreover, the new constraint in Equation (11) can be approximated through popular risk measures such as the Value at Risk and Conditional Value at Risk. Illustrated in Figure 2, these two risk measures are formally expressed as follows:

$${\mathrm{VaR}}_{\rho }\left(Z^{\pi }\right)=\inf \left\{z\in \mathbb{R}:F_{Z^{\pi }}\left(z\right)\ge \rho \right\},$$

(12)

$${\mathrm{CVaR}}_{\rho }\left(Z^{\pi }\right)=\mathbb{E}\left[z|z\le {\mathrm{VaR}}_{\rho }\left(Z^{\pi }\right)\right],$$

(13)

where $F_{Z^{\pi }}$ represents the CDF of the random return $Z^{\pi }$.

More generally, the present research work introduces the state-action risk function $R^{\pi }:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ of a decision-making policy $\pi$, which is the equivalent of the Q function for the risk. More precisely, that function $R^{\pi }(s,a)$ quantifies the riskiness of the discounted sum of rewards obtained by executing an action $a\in \mathcal{A}$ in a state $s\in \mathcal{S}$ and then following policy $\pi$:

$$R^{\pi }(s,a)={\mathcal{R}}_{\rho }\left[Z^{\pi }(s,a)\right],$$

(14)

where:

•

${\mathcal{R}}_{\rho }:\mathcal{Z}\to \mathbb{R}$ is a function extracting risk features from the random return probability distribution $Z^{\pi }$, such as ${\mathrm{VaR}}_{\rho }$ or ${\mathrm{CVaR}}_{\rho }$,

•

$\rho \in \left[0,1\right]$ is a parameter corresponding to the cumulative probability associated with the worst returns, generally between $0\%$ and $10\%$. In other words, this parameter controls the size of the random return distribution tail from which the risk is estimated.

3.2.3. Risk-Based Utility Function

In order to pursue the objective criterion defined in Equation (11) for risk-sensitive RL, this research work introduces a new concept: the state-action risk-based utility function. Denoted $U^{\pi }(s,a)$, the utility function assesses the quality of an action $a\in \mathcal{A}$ in a certain state $s\in \mathcal{S}$ in terms of expected performance and risk, assuming that the policy $\pi$ is followed afterwards. In fact, the intent is to extend the popular Q function so that the risk is taken into consideration by taking advantage of the risk function defined in Section 3.2.2. More precisely, the utility function $U^{\pi }$ is built as a linear combination of the $Q^{\pi }$ and $R^{\pi }$ functions. Formally, the risk-based utility function $U^{\pi }:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ of a policy $\pi$ is defined as the following:

$$\begin{array}{cc}\hfill U^{\pi }(s,a)& =\alpha Q^{\pi }(s,a)+(1-\alpha )R^{\pi }(s,a)\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & =\alpha \mathbb{E}\left[Z^{\pi }(s,a)\right]+(1-\alpha ){\mathcal{R}}_{\rho }\left[Z^{\pi }(s,a)\right],\hfill \end{array}$$

(16)

where $\alpha \in \left[0,1\right]$ is a parameter controlling the trade-off between expected performance and risk. If $\alpha =0$, the utility function will be maximised with a fully risk-averse decision-making policy. On the contrary, if $\alpha =1$, the utility function degenerates into the Q function quantifying the performance on expectation. Figure 3 graphically describes the utility function $U^{\pi }$, which moves along the x-axis between the quantities $R^{\pi }$ and $Q^{\pi }$ when modifying the value of the parameter $\alpha$.

3.2.4. Risk-Sensitive Distributional RL Algorithm

In most applications, the motivation for choosing the distributional RL approach over the classical one is related to the improved expected performance that results from learning a richer representation of the environment. Despite having access to the full probability distribution of the random return, only the expectation is exploited to derive decision-making policies:

$$\pi \left(s\right)\in \underset{a\in \mathcal{A}}{argmax}\underset{Q^{\pi }(s,a)}{\underbrace{\mathbb{E}\left[Z^{\pi }(s,a)\right]}}.$$

(17)

Nevertheless, as previously hinted, the random return $Z^{\pi }$ also contains valuable information about the risk, which could be exploited to learn risk-sensitive decision-making and exploration policies. The present research work suggests that risk-sensitive distributional RL should be achieved by maximising the utility function $U^{\pi }$, derived from $Z^{\pi }$, instead of the expected return $Q^{\pi }$ when selecting actions. This alternative operation would be performed during both exploration and exploitation. Even though maximising the utility function is not exactly equivalent to the optimisation of the objective criterion defined in Equation (11), it is a relevant step towards risk-sensitive RL. Consequently, a risk-sensitive policy $\pi$ can be derived as follows:

$$\pi \left(s\right)\in \underset{a\in \mathcal{A}}{argmax}{U}^{\pi }(s,a).$$

(18)

Throughout the learning phase, a classical Q-learning algorithm is expected to progressively converge towards the optimal value function $Q^{*}$ that naturally arises from the optimal policy ${\pi }^{*}$. In a similar way, the proposed risk-sensitive RL algorithm jointly learns the optimal policy ${\pi }^{*}$ and the optimal state-action risk-based utility function $U^{*}$. More formally, the latter two are mathematically defined as the following:

$$U^{*}(s,a)=\alpha \mathbb{E}\left[Z^{*}(s,a)\right]+(1-\alpha ){\mathcal{R}}_{\rho }\left[Z^{*}(s,a)\right],$$

(19)

$$Z^{*}(s,a)\stackrel{D}{=}R(s,a)+\gamma Z^{*}\left(s^{\prime },{\pi }^{*}\left(s^{\prime }\right)\right),$$

(20)

$${\pi }^{*}\left(s\right)\in \underset{a\in \mathcal{A}}{argmax}{U}^{*}(s,a).$$

(21)

The novel methodology proposed by this research work to learn risk-sensitive decision-making policies based on the distributional RL approach is summarised as follows. Firstly, select any distributional RL algorithm that learns the full probability distribution of the random return $Z^{\pi }$. Secondly, leave the learning process unchanged except for action selection, which involves the maximisation of the utility function $U^{\pi }$ rather than the expected return $Q^{\pi }$. This adaptation is the single change to the distributional RL algorithm, occurring at two different locations within the algorithm: i. the generation of new experiences by interacting with the environment, ii. the learning of the random return $Z^{\pi }$ based on the distributional Bellman equation. However, this adaptation has no consequence on the random return $Z^{\pi }$ learnt by the distributional RL algorithm, only on the actions derived from that probability distribution. Algorithm 1 details the proposed solution in a generic way, with the required modifications highlighted.

3.3. Performance Assessment Methodology

3.3.1. Benchmark Environments

This research work introduces some novel benchmark environments in order to assess the soundness of the proposed methodology to design risk-sensitive policies based on the distributional RL approach. These environments consist of three toy problems that are specifically designed to highlight the importance of considering the risk in a decision-making policy. More precisely, the control problems are built in such a way that the optimal policy will differ depending on whether the objective is to solely maximise the expected performance or to also mitigate the risk. This is achieved by including relevant stochasticity in both the state transition function $p_T$ and reward function $p_R$. Moreover, the benchmark environments are designed to be relatively simple in order to ease the analysis and understanding of the decision-making policies tha tare learnt. This simplicity also ensures the accessibility of the experiments, since distributional RL algorithms generally require a considerable amount of computing power. Figure 4 illustrates these three benchmark environments, and highlights the optimal paths to be learnt depending on the objective being pursued. For the sake of completeness, a thorough description of the underlying MDPs is provided in Appendix A, including the stochastic reward and transition functions.

Algorithm 1 Risk-sensitive distributional RL algorithm.

Initialise the experience replay memory M of capacity C. Initialise both the main and target DNN weights $\theta ={\theta }^{-}$. for episode = 0 to N do    for time step $t=0$ to T, or until episode termination do      Acquire the state s from the environment $\mathcal{E}$.      With probability $ϵ$, select a random action $a\in \mathcal{A}$.      Otherwise, select the action $a={argmax}_{a^{\prime }\in \mathcal{A}}{U}^{\pi }(s,a^{\prime };\theta )$.      Execute action a in environment $\mathcal{E}$ to get the next state $s^{\prime }$ and the reward r.      Store the experience $e=(s,a,r,s^{\prime })$ in M.      Randomly sample from M a minibatch of $N_e$ experiences $e_i=(s_i,a_i,r_i,s_i^{{}^{\prime }})$.      for $i=0$ to $N_e$ do         Distributional Bellman equation:                 $Z^{\pi }(s_i,a_i;\theta )\stackrel{D}{=}{r}_i+\gamma Z^{\pi }(s_{i+1},{argmax}_{a_i^{\prime }\in \mathcal{A}}{U}^{\pi }(s_{i+1},a_i^{\prime };{\theta }^{-});{\theta }^{-})$.      end for      Compute the resulting loss $\mathcal{L}\left(\theta \right)$, according to the probability metric selected.      Update the main DNN parameters $\theta$ using a deep learning optimiser with learning rate $L_r$.      Update the target DNN parameters ${\theta }^{-}=\theta$ every $N^{-}$ steps.      Anneal the $ϵ$-greedy exploration parameter $ϵ$.    end for end for

The first benchmark environment is named risky rewards. It consists of a $3\times 3$ grid world, within which an agent has to reach one of two objective areas that are equidistant from its fixed initial location. The difficulty of this control problem lies in the choice of the objective area to target, because of the stochasticity present in the reward function. Reaching the first objective area yields a reward with a lower value in expectation and a limited deviation from that average. On the contrary, reaching the second objective location yields a reward that is higher in expectation, at the cost of an increased risk.

The second benchmark environment that was studied is named risky transitions. It consists of a $3\times 3$ grid world, within which an agent has to reach one of two objective areas as quickly as possible in the presence of a stochastic wind. The agent is initially located in a fixed area that is very close to an objective, but the required move to reach it is in opposition to the wind direction. Following that path results in a reward that is higher in expectation, but there is a risk to be repeatedly countered, caused by the stochastic wind. On the contrary, the longer path is safer but yields a lower reward on average.

The last presented benchmark environment is named risky grid world. This control problem can be viewed as a combination of the two previously described environments since it integrates both stochastic rewards and transitions. It consists of a $3\times 3$ grid world within which an agent, initially located in a fixed area, has to reach a fixed objective location as quickly as possible. To achieve that goal, three paths are available. The agent may choose the shortest path to the objective location, characterised by a stochastic trap, or get around this risky situation by taking a significantly longer route. This bypass can be made from the left or from the right, another critical choice in terms of risk because of the stochastic wind. Once again, the optimal path is dependent on the objective criterion being pursued.

3.3.2. Risk-Sensitive Distributional RL Algorithm Analysed

The distributional RL algorithm selected to assess the soundness of the methodology introduced to learn risk-sensitive decision-making policies is the Unconstrained Monotonic Deep Q-Network with Cramer (UMDQN-C) [27]. Basically, this particular distributional RL algorithm models the CDF of the random return in a continuous way by taking advantage of the Cramer distance to derive the TD-error. Moreover, the probability distributions learnt are ensured to be valid thanks to the specific architecture exploited to model the random return: Unconstrained Monotonic Neural Network (UMNN) [28]. The latter has been demonstrated to be a universal approximator of continuous monotonic functions, which is particularly convenient when representing CDFs. In practice, the UMDQN-C algorithm has been shown to achieve great results, both in terms of policy performance and in terms of random return probability distribution quality. This second feature clearly motivated the selection of this specific distributional RL algorithm to conduct the following experiments, since accurate random return probability distributions are required to properly estimate the risk. The reader can refer to the original research paper [27] for more information about the UMDQN-C distributional RL algorithm.

As previously explained in Section 3.2.2, the presented approach requires the choice of a function ${\mathcal{R}}_{\rho }$ to extract risk features from the random return probability distribution $Z^{\pi }$. In the following experiments, Value at Risk (VaR) is adopted to estimate the risk. This choice is motivated by both the popularity of that risk measure in practice and by the efficiency of computation. Indeed, this quantity can be conveniently derived from the CDF of the random return learnt by the UMDQN-C algorithm.

In the next section presenting the achieved results, the risk-sensitive version of the UMDQN-C algorithm is denoted RS-UMDQN-C. The detailed pseudo-code of that new risk-sensitive distributional RL algorithm can be found in Appendix B.

To conclude this section, ensuring the reproducibility of the results in a transparent way is particularly important to this research work. In order to achieve this,

Table 1. Description of the main hyperparameters used in the experiments.

Hyperparameter	Symbol	Value
DNN structure	-	$[128,128]$
Learning rate	$L_r$	${10}^{-4}$
Deep learning optimiser epsilon	-	${10}^{-5}$
Replay memory capacity	C	${10}^4$
Batch size	$N_e$	32
Target update frequency	$N^{-}$	${10}^3$
Random return resolution	$N_z$	200
Random return lower bound	$z_{\min }$	$-2$
Random return upper bound	$z_{\max }$	$+2$
Exploration $ϵ$-greedy initial value	-	$1.0$
Exploration $ϵ$-greedy final value	-	$0.01$
Exploration $ϵ$-greedy decay	-	${10}^4$
Risk coefficient	$\rho$	$10\%$
Risk trade-off	$\alpha$	$0.5$

provides a brief description of the key hyperparameters used in the experiments. Moreover, the entire experimental code is made publicly available at the following link: https://github.com/ThibautTheate/Risk-Sensitive-Policy-with-Distributional-Reinforcement-Learning, accessed on 10 June 2023.

4. Results and Discussion

4.1. Decision-Making Policy Performance

To begin with, the performance achieved by the decision-making policies $\pi$ learnt has to be evaluated, both in terms of expected outcome and risk. For comparison purposes, the results obtained by the well-established DQN algorithm, a reference without any form of risk sensitivity, are presented alongside those of the newly introduced RS-UMDQN-C algorithm. It should be mentioned that these two RL algorithms achieve very similar results when risk sensitivity is disabled ($\alpha =1$ for the RS-UMDQN-C algorithm), as expected.

In the following, two analyses are presented. Firstly, the probability distribution of the cumulative reward of a policy $\pi$, denoted $S^{\pi }\in \mathcal{Z}$, is investigated. More precisely, the expectation $\mathbb{E}[·]$, the risk function ${\mathcal{R}}_{\rho }[·]$ and the utility function $U[·]$ of that random variable $S^{\pi }$ are derived for each algorithm and compared. Secondly, this research work introduces a novel easy-to-interpret performance indicator $R_s\in [-1,1]$ to evaluate the risk sensitivity of the decision-making policies that are learnt, by taking advantage of the simplicity of the benchmark environments presented in Section 3.3.1. In fact, this is made possible by the easy assessment, from a human perspective, of the relative riskiness of a path in the grid world environments being studied. If the optimal path in terms of risk is chosen (green arrows in Figure 4), a score $R_s=+1$ is awarded. On the contrary, the riskier path, which is optimal in terms of expectation (orange arrows in Figure 4) yields a score of $R_s=-1$. If no objective or trap areas are reached within the allowed time, a score of $R_s=0$ is delivered. Consequently, the evolution of this performance indicator provides valuable information about the convergence of the RL algorithms towards the different possible paths, as well as about the stability of the learning process. Formally, let $\tau ={\left\{s_t,a_t\right\}}_{t\in [0,T]}$ with $s_t\in \mathcal{S}$ and $a_t\in \mathcal{A}$ be a trajectory defined over a time horizon $T<10$ (ending with a terminal state, and subject to an upper bound), and let ${\tau }_{+}$ and ${\tau }_{-}$, respectively, be the sets of trajectories associated with the green and orange paths in Figure 4. Based on these definitions, the risk sensitivity $R_s$ of a policy $\pi$ is a random variable that can be assessed via Monte Carlo as the following:

$$R_s\left(\pi \right)=\left\{\begin{array}{cc}+1\hfill & \mathrm{if}\pi \mathrm{produces}\mathrm{trajectories}{\left\{s_t,a_t\right\}}_{t\in [0,T]}\in {\tau }_{+}\mathrm{with}{a}_t=\pi \left(s_t\right),\hfill \\ -1\hfill & \mathrm{if}\pi \mathrm{produces}\mathrm{trajectories}{\left\{s_t,a_t\right\}}_{t\in [0,T]}\in {\tau }_{-}\mathrm{with}{a}_t=\pi \left(s_t\right),\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(22)

The first results regarding policy performance are summarised in

Table 2. Comparison of the expectation $\mathbb{E}[·]$, the risk function ${\mathcal{R}}_{\rho }[·]$ and the utility function $U[·]$ of the cumulative reward $S^{\pi }$ achieved by the decision-making policies $\pi$ learnt by both the DQN and RS-UMDQN-C algorithms.

Benchmark Environment	DQN			RS-UMDQN-C
	$\mathbb{E}\left[{\mathit{S}}^{\mathit{\pi }}\right]$	${\mathcal{R}}_{\mathit{\rho }}\left[{\mathit{S}}^{\mathit{\pi }}\right]$	$\mathit{U}\left[{\mathit{S}}^{\mathit{\pi }}\right]$	$\mathbb{E}\left[{\mathit{S}}^{\mathit{\pi }}\right]$	${\mathcal{R}}_{\mathit{\rho }}\left[{\mathit{S}}^{\mathit{\pi }}\right]$	$\mathit{U}\left[{\mathit{S}}^{\mathit{\pi }}\right]$
Risky rewards	0.3	−1.246	−0.474	0.1	−0.126	−0.013
Risky transitions	0.703	0.118	0.411	0.625	0.346	0.485
Risky grid world	0.347	−1.03	−0.342	0.333	0.018	0.175

, which compares the decision-making policies learnt by the DQN and RS-UMDQN-C algorithms both in terms of expected outcome and risk. The second results regarding policy performance are compiled in Figure 5, plotting the evolution of the risk-sensitivity performance indicator $R_s$ during the training phase. It can be clearly observed from these two analyses that the proposed approach is effective in learning decision-making policies that are sensitive to the risk for relatively simple environments. As expected, the DQN algorithm yields policies that are optimal in expectation, whatever the level of risk incurred. In contrast, the RS-UMDQN-C algorithm is able to leverage both expected outcome and risk in order to learn decision-making policies that produce a slightly lower expected return with a significantly lower risk level. This allows for the proposed methodology to significantly outperform the risk-neutral RL algorithm of reference with respect to the performance indicator of interest $U\left[S^{\pi }\right]$ in Table 2. Finally, it is also encouraging to observe from Figure 5 that the learning process seems to be quite stable for simple environments, despite having to maximise a much more complicated function.

4.2. Probability Distribution Visualisation

A core advantage of the proposed solution is the improved interpretability of the resulting decision-making process. Indeed, the understanding and motivation of the decisions outputted by the learnt policy $\pi$ is greatly facilitated by the access to the probability distributions of the random return jointly learnt. In addition, the analysis and comparison of the value, risk and utility functions ($Q^{\pi }$, $R^{\pi }$ and $U^{\pi }$) associated with different actions provide a valuable summary of the decision-making process, but also about the control problem itself. Such an analysis may be particularly important to correctly tune the risk trade-off parameter $\alpha$ according to the user’s risk aversion.

As an illustration, Figure 6 demonstrates some random return probability distributions $Z^{\pi }$ that are learnt by the RS-UMDQN-C algorithm. More precisely, a single relevant state is selected for analysis for each benchmark environment. The selection is based on the importance of the next decision in following a clear path, either maximising the expected outcome or mitigating the risk. Firstly, it can be observed that the risk-sensitive distributional RL algorithm manages to accurately learn the probability distributions of the random return, qualitatively, from a human perspective. In particular, the multimodality purposely designed to create risky situations appears to be well-preserved. Such a result is particularly encouraging, since this feature is essential to the success of the proposed methodology. Indeed, it ensures the accurate estimation of the risk, as defined in Section 3.2.2. This observation is in line with the findings of the research paper [27], introducing the UMDQN algorithm, and suggests that the solution introduced to achieve risk-sensitivity does not significantly alter the properties of the original distributional RL algorithm. Secondly, as previously explained, Figure 6 highlights the relevance of each function introduced ($Q^{\pi }$, $R^{\pi }$ and $U^{\pi }$) for making and motivating a decision. Their analysis truly contributes to the understanding of the potential trade-off between expected performance maximisation and risk mitigation for a given decision-making problem, as well as the extent to which different values of the important parameter $\alpha$ lead to divergent policies.

5. Conclusions

The present research work introduces a straightforward yet efficient solution to learning risk-sensitive decision-making policies based on the distributional RL approach. The proposed methodology presents key advantages. Firstly, it is perfectly compatible with any distributional RL algorithm, and requires only minimal modifications to the original algorithm. Secondly, the simplicity of the approach contributes to the interpretability and ease of analysis of the resulting risk-sensitive policies, a particularly important feature to avoid black-box machine learning models. Lastly, the solution can cover the complete potential trade-off between expected outcome maximisation and risk minimisation. The first experiments performed on three relevant toy problems yield promising results, which may be viewed as a proof of concept for the accessible and practical solution introduced.

Some interesting leads can be suggested as future work. Firstly, the research that was conducted is exclusively empirical and does not study any theoretical guarantees about the resulting risk-sensitive distributional RL algorithms. Among others, the study of the convergence of these algorithms would be a relevant future research direction. Secondly, building on the promising results achieved, the presented solution should definitely be evaluated on more complex environments, for which the risk should ideally be mitigated. Lastly, the approach could be extended to not only mitigate the risk but also to completely discard actions that would induce an excessive level of risk in order to increase compliance with the objective criterion originally defined in Section 3.2.1.