Best-Arm Identification Using Extreme Value Theory Estimates of the CVaR

Abstract

We consider a risk-aware multi-armed bandit framework with the goal of avoiding catastrophic risk. Such a framework has multiple applications in financial risk management. We introduce a new conditional value-at-risk (CVaR) estimation procedure combining extreme value theory with automated threshold selection by ordered goodness-of-fit tests, and we apply this procedure to a pure exploration best-arm identification problem under a fixed budget. We empirically compare our results with the commonly used sample average estimator of the CVaR, and we show a significant performance improvement when the underlying arm distributions are heavy-tailed.

1. Introduction

In the stochastic multi-armed bandit (MAB) problem, a learning agent is presented with the repeated task of selecting from a number of choices (arms), each providing independent and identically distributed rewards/costs. The agent has no prior knowledge of the underlying arm distributions. Through a combination of exploration and exploitation, the agent attempts to identify the arm with the most favorable reward/cost distribution as quickly as possible; see Lattimore and Szepesvári (2020) for a description of such a setting, which has many applications, such as these detailed in Bouneffouf et al. (2020).

In the traditional MAB framework, the most favorable distribution maximizes the expected reward. Recent generalizations of this problem replace the expectation objective with other metrics aimed at measuring risk. For example, Bhat and Prashanth (2019); David et al. (2018); Galichet et al. (2013); Kagrecha et al. (2019); Sani et al. (2012); Torossian et al. (2019); Yu and Nikolova (2013) addressed the MAB problem with a risk-averse (risk-aware) agent.

Such a framework is better suited to applications in finance where a risk-reward trade-off is sought rather than pure profit maximization. Example of applications of risk-aware multi-armed bandits in finance include portfolio optimization as in Hoffman et al. (2011); Shen et al. (2015); Moeini et al. (2016) and Huo and Fu (2017), cryptocurrency investing as in Yu et al. (2018), hedging as in Cannelli et al. (2020) and energy markets trading as in Costa and Oliveira (2008) and Ariffin et al. (2016).

Often, the risk measure of interest is the conditional value-at-risk (CVaR). Given a continuous random variable X representing a cost (i.e., where larger values are less desirable), the CVaR at a confidence level $\alpha \in (0,1)$ measures the expected value of X given that X exceeds the quantile at level $\alpha .$ This quantile is referred to as the value-at-risk (VaR). Compared to the VaR, the CVaR captures more information about the weight of a distribution’s tail, making it a more useful object of study in risk-averse decision making.

In previous papers on the risk-averse MAB problem, the CVaR is usually estimated by averaging observations above the estimated VaR. When $\alpha$ is close to 1, these observations can be very scarce in small samples, leading to imprecise estimates. This is particularly apparent in heavy-tailed distributions, where extreme events correspond to very large values that are not readily observed. This is in contrast to light-tailed distributions, where similar low probability events are closer to the mean and not as catastrophic.

In this paper, we seek to address the CVaR estimation problem described above within the MAB framework, which has applications in any scenario where the primary goal of the agent is to avoid a severely unfavourable outcome, such as financial ruin, patient deaths in clinical trials or the adverse impact of a natural disaster. The MAB problem with heavy-tailed underlying arm distributions has been analyzed in, for example, Bubeck et al. (2013) and Prashanth et al. (2020). As in the latter, we consider the problem of identifying the arm with the lowest CVaR value in a fixed budget (pure exploration) MAB problem.

This paper considers the CVaR for single stages at a time, i.e., not the CVaR on aggregates of rewards across various stages. Instead of estimating the CVaR by sample averaging, we consider an alternative estimator based on extreme value theory (EVT), e.g.,  (McNeil et al. 2005, Section 7.2.3). EVT has numerous applications in risk management (see for instance Embrechts et al. 1999 and Embrechts et al. 2013), and the present work is an attempt to incorporate its benefits to the MAB problem,  reinforcement learning and the machine learning literature.

By selecting a threshold lower than the VaR, it is possible to approximate the tail distribution of a random variable by using a generalized Pareto distribution (GPD) and extrapolating beyond the available observations. The major difficulty of this approach is the bias-variance tradeoff that results from threshold selection: too high of a threshold results in limited data availability causing high variance, whereas too low of a threshold can cause a large bias between the GPD and true tail distribution.

Traditionally, threshold selection methods often rely on the visual inspection of graphical plots and apply to a fixed dataset, making them unsuitable for the online learning framework of the MAB problem. Recently, the work of Bader et al. (2018) presented a solution to threshold selection by automating the process using ordered goodness-of-fit tests. This method has recently shown promising experimental results in VaR estimation, i.e., Zhao et al. (2018), and we propose to use this method in an algorithm for the CVaR in MABs.

The main contribution of this paper is the design of a CVaR estimation algorithm that combines the EVT method with automated threshold selection and application of this algorithm in the MAB context. Such an algorithm is crucial (typo) from a practical standpoint since it allows rigorously tackling risk-aware MAB problems requiring avoiding catastrophic risk, a class of problems for which the existing methodologies are ill-adapted due to the scarcity of data related to extreme outcomes. The present paper also bridges the gap between the MAB literature and the extreme value theory one which, to the best of the authors knowledge, have never been previously combined. In our simulation study, this estimation procedure outperformed the commonly used sample average estimator in heavy-tailed distributions.

The remainder of this paper is organized as follows. In Section 2, the VaR and CVaR are formally defined, and the sample average estimator of the CVaR is given. Needed background from EVT is discussed, and we introduce approximate formulas for the VaR and CVaR. We present our CVaR estimation procedure using EVT and automated threshold selection in Section 3, and we introduce the risk-averse successive rejects algorithm for best-arm identification in Section 4. Numerical experiments are presented in Section 5, and we conclude in Section 6.

2. Preliminaries

2.1. Risk Measures

Let X denote a random variable and F its corresponding cumulative distribution function (CDF). In this paper, we adopt the convention that X represents a cost to the agent, and thus larger values of X are less desirable.

Definition 1

(Value-at-Risk). The value-at-risk of X at level $\alpha \in (0,1)$ is

$${\mathrm{VaR}}_{\alpha }\left(X\right)=inf\{x\in \mathbb{R}|F\left(x\right)\ge \alpha \}.$$

(1)

${\mathrm{VaR}}_{\alpha }\left(X\right)$ is equivalent to the quantile at level $\alpha$ of F. If the inverse of F exists, ${\mathrm{VaR}}_{\alpha }\left(X\right)=F^{-1}\left(\alpha \right)$. The VaR can be estimated in the same way as the standard empirical quantile. Let $X_1,\dots ,X_n$ be i.i.d. random variables with common CDF F. Denote by $X_{(1,n)}\le X_{(2,n)}\dots \le X_{(n,n)}$ the set of order statistics for the sample of size n, i.e., the sample sorted in non-decreasing order. An estimator for the VaR is

$${\widehat{\mathrm{VaR}}}_{n,\alpha }\left(X\right)=min\left\{X_{(i,n)}|i=1,\dots ,n;{\widehat{F}}_n\left(X_{(i,n)}\right)\ge \alpha \right\}=X_{(m,n)},$$

where ${\widehat{F}}_n$ denotes the empirical CDF and $m=\lceil \alpha n\rceil$ with $\lceil x\rceil$ denoting the smallest integer greater or equal to x. Such an estimate corresponds to the $\alpha$-level VaR of the empirical distribution of X based on the sample of size n. We now define the CVaR as in Acerbi and Tasche (2002).

Definition 2

(Conditional Value-at-Risk). The conditional value-at-risk of a continuous random variable X at level $\alpha \in (0,1)$ is

$${\mathrm{CVaR}}_{\alpha }\left(X\right)=\mathbb{E}\left[X\right|X\ge {\mathrm{VaR}}_{\alpha }\left(X\right)]=\frac{1}{1-\alpha }{\int }_{\alpha }^1{\mathrm{VaR}}_{\gamma }\left(X\right)d\gamma .$$

(2)

Typical values of $\alpha$ are $0.95$, $0.99$, $0.999$, etc. Without loss of generality, the current work only considers continuous random variables. For simplicity of notation, we refer to ${\mathrm{CVaR}}_{\alpha }\left(X\right)$ and ${\mathrm{VaR}}_{\alpha }\left(X\right)$ by $c_{\alpha }$ and $q_{\alpha }$, respectively, for the remainder of this paper. Typically, the CVaR is estimated by averaging observations above ${\widehat{\mathrm{VaR}}}_{\alpha }\left(X\right)$. This estimator is given by

$${\widehat{\mathrm{CVaR}}}_{n,\alpha }\left(X\right)=\frac{{\sum }_{i=1}^n{X}_i{𝟙}_{\{X_i\ge {\widehat{\mathrm{VaR}}}_{n,\alpha }\left(X\right)\}}}{{\sum }_{j=1}^n{𝟙}_{\{X_j\ge {\widehat{\mathrm{VaR}}}_{n,\alpha }\left(X\right)\}}},$$

(3)

which would coincide with $\mathbb{E}\left[Z\right|Z\ge {\mathrm{VaR}}_{\alpha }\left(Z\right)]$ under the assumption that the distribution of Z is the empirical distribution of X. The CVaR risk measure has proven extremely popular in the risk management literature due to its favorable theoretical properties. For instance, it is a coherent risk measure as defined by Artzner et al. (1999), since it satisfies the monotonicity, translation invariance, subadditivity and positive homogeneity properties.

The use of Equation (3) can be problematic when the confidence level $\alpha$ is high due to the scarcity of extreme observations. We now provide tools from extreme value theory to address this problem.

2.2. Extreme Value Theory and Heavy Tails

Let $F^n\left(x\right)=\mathbb{P}(max(X_1,\dots ,X_n)\le x)$ denote the CDF of the sample maximum. Suppose there exists a sequence of real-valued constants $a_n>0$ and $b_n$, $n=1,2,\dots$, and a nondegenerate CDF H such that

$$\underset{n\to \infty }{lim}{F}^n\left(a_{n}x+b_n\right)=H\left(x\right),$$

(4)

for all x, where nondegenerate refers to a distribution not concentrated at a single point. The class of distributions F that satisfy (4) are said to be in the maximum domain of attraction of H, denoted $F\in MDA\left(H\right)$. The Fisher–Tippett–Gnedenko theorem (see De Haan and Ferreira (2006, Theorem 1.1.3)) states that H must then be a generalized extreme value distribution (GEVD), which is given in the following definition.

Definition 3

(GEVD). The generalized extreme value distribution (GEVD) with single parameter $\xi \in \mathbb{R}$ has the CDF

$$H_{\xi }\left(x\right)=\left\{\begin{array}{c}exp\left(-{(1+\xi x)}^{-1/\xi }\right)\mathit{if}\xi \ne 0,\hfill \\ exp\left(-e^{-x}\right)\mathit{if}\xi =0,\hfill \end{array}\right.$$

over its support, which is $[-1/\xi ,\infty )$ if $\xi >0$, $(-\infty ,-1/\xi ]$ if $\xi <0$ or $\mathbb{R}$ if $\xi =0$.

If $F\in MDA\left(H\right)$, then there exists a unique $\xi \in \mathbb{R}$ such that $H=H_{\xi }$. It is important to note that essentially all common continuous distributions used in applications are in MDA$\left(H_{\xi }\right)$ for some value of $\xi$, and this value characterizes the speed of the tail decay. When $\xi >0$, then F is a heavy-tailed distribution, which means that moments of order greater than or equal to $1/\xi$ do not exist. The tail decreases like a power law, and very large observations can occur with a non-negligible probability. Otherwise, F is light-tailed with a tail with exponential decay ($\xi =0$), or the right endpoint of F is finite ($\xi <0$). If $\xi \ge 1$, then F has infinite mean, and therefore the true CVaR, Equation (2), is also infinite. For the remainder of this paper, we assume that the following condition is satisfied.

Assumption 1.

$F\in MDA\left(H_{\xi }\right)$ with $\xi \in [0,1)$.

When $F\in MDA\left(H_{\xi }\right)$, there exists a useful approximation of the distribution of sample extremes above a high threshold.

Definition 4

(Excess distribution function). For a given threshold $u>essinfX$, the excess distribution function is defined as

$$F_u\left(y\right)=\mathbb{P}(X-u\le y|X>u)=\frac{F(y+u)-F\left(u\right)}{1-F\left(u\right)},y>0.$$

The y-values are referred to as the threshold excesses. Given that X has exceeded some high threshold u, this function represents the probability that X exceeds the threshold by at most y. For large values of u, the Pickands–Balkema–de Haan theorem states that $F_u$ can be well-approximated by the GPD. We recall this theorem now.

Theorem 1

(Balkema and De Haan 1974; Pickands III et al. 1975). Suppose Assumption 1 is satisfied. Then, there exists a positive function $\sigma =\sigma \left(u\right)$ such that

$$\underset{u\to \infty }{lim}\underset{0\le y\le \infty }{sup}|F_u\left(y\right)-G_{\xi ,\sigma }\left(y\right)|=0,$$

(5)

where $G_{\xi ,\sigma }$ is the CDF of the generalized Pareto distribution, given by

$$G_{\xi ,\sigma }\left(y\right)=\left\{\begin{array}{c}1-{\left(1+\frac{\xi y}{\sigma }\right)}^{-1/\xi },\xi \ne 0,\hfill \\ 1-e^{-y/\sigma },\xi =0,\hfill \end{array}\right.$$

(6)

over its support, which is $[0,\infty )$ if $\xi \ge 0$ or $[0,-\sigma /\xi ]$ if $\xi <0$.

Using Theorem 1, it is quite straightforward to derive approximate formulas for the VaR and CVaR using the definition of the excess CDF and Equations (1) and (2); for example, see McNeil et al. (2005, Section 7.2.3).

Corollary 1

(EVT approximations). Suppose that Assumption 1 is satisfied. Fix $u>essinfX$ and let $\sigma =\sigma \left(u\right)$ be a function satisfying Equation (5). Then, the EVT approximations for the VaR and CVaR at level $\alpha >F\left(u\right)$ are given by, respectively,

$$q_{u,\alpha }=\left\{\begin{array}{c}u+\frac{\sigma }{\xi }\left(s_{u,\alpha }^{\xi }-1\right),\xi \ne 0,\hfill \\ u+\sigma \log s_{u,\alpha },\xi =0,\hfill \end{array}\right.$$

(7)

$$c_{u,\alpha }=\left\{\begin{array}{c}u+\frac{\sigma }{1-\xi }\left(1+\frac{s_{u,\alpha }^{\xi }-1}{\xi }\right),\xi \ne 0,\hfill \\ u+\sigma \left(\log s_{u,\alpha }+1\right),\xi =0,\hfill \end{array}\right.$$

(8)

where $s_{u,\alpha }=(1-F\left(u\right))/(1-\alpha )$.

The accuracy of the EVT approximations depends on how high of a threshold is used. When these approximations are used in statistical estimation, a lower threshold is preferable to make use of as much data as possible; however, this can induce a significant bias. In the next section, we introduce our algorithm for estimating the CVaR with the EVT approximations and automated threshold selection.

3. Estimating the CVaR through EVT

3.1. Maximum Likelihood Estimation of the GPD Parameters

In this section, we discuss the estimation of $c_{u,\alpha }$ using Corollary 1 and the maximum likelihood. This can be done by first selecting a threshold u and then estimating the GPD parameters using the threshold excesses above u. For brevity, we only present the maximum likelihood estimators (MLEs) for $\xi \ne 0$; for the $\xi =0$ case, see De Haan and Ferreira (2006, Section 3.4).

For an i.i.d. sample $X_1,\dots X_n$, the threshold excesses are given by $\{X_i-u|i=1,\dots ,n;X_i>u\}$, and we denote them as $Y_1,\dots ,Y_k$. Assuming $k>0$, the MLEs are obtained by maximizing the approximate log-likelihood function with respect to $\xi$ and $\sigma$,

$$\begin{array}{c}\hfill ({\widehat{\xi }}_u^{\left(n\right)},{\widehat{\sigma }}_u^{\left(n\right)})=\underset{\xi ,\sigma }{arg\; max}\sum _{i=1}^k\log g_{\xi ,\sigma }\left(Y_i\right),\end{array}$$

(9)

where $(\xi ,\sigma )\in (-1/2,\infty )\times (0,\infty )$ and $g_{\xi ,\sigma }$ is the probability density function of the GPD, which, for $\xi \ne 0$, is given by

$$g_{\xi ,\sigma }\left(y\right)=\frac{1}{\sigma }{\left(1+\frac{\xi y}{\sigma }\right)}^{-1/\xi -1}.$$

Based on partial derivatives of the log-pdf with respect to parameters, the resulting maximum likelihood first-order conditions are given by

$$\frac{1}{k}\sum _{i=1}^k\log \left(1+\frac{\xi Y_i}{\sigma }\right)=\xi ,\frac{1}{k}\sum _{i=1}^k\frac{Y_i}{\sigma +\xi Y_i}=\frac{1}{\xi +1}.$$

(10)

A closed-form solution to Equation (10) does not exist; however, the MLEs can be obtained numerically through standard software packages, e.g., scipy.stats.genpareto in Python.

Using Equation (8) with the MLEs and an appropriate selection of threshold leads to the following estimator of the CVaR.

Definition 5

(EVT CVaR estimator). Suppose that $({\widehat{\xi }}_u^{\left(n\right)},{\widehat{\sigma }}_u^{\left(n\right)})$ are obtained from the excesses above a chosen threshold u. Then, for $\alpha >{\widehat{F}}_n\left(u\right)$, an estimator for the CVaR is

$${\widehat{c}}_{u,\alpha }^{\left(n\right)}\equiv u+\frac{{\widehat{\sigma }}_u^{\left(n\right)}}{1-{\widehat{\xi }}_u^{\left(n\right)}}\left(1+\frac{1}{{\widehat{\xi }}_u^{\left(n\right)}}\left[{\left(\frac{1-{\widehat{F}}_n\left(u\right)}{1-\alpha }\right)}^{{\widehat{\xi }}_u^{\left(n\right)}}-1\right]\right),$$

(11)

with a similar interpretation based on the second line of Equation (8) when ${\widehat{\xi }}_u^{\left(n\right)}=0$.

Note that estimates $({\widehat{\xi }}_u^{\left(n\right)},{\widehat{\sigma }}_u^{\left(n\right)})$ exhibit asymptotic bias as explained in Troop et al. (2021). The estimation procedure from the latter paper allowing to remove asymptotic bias could be explored in the context of multi-bandits in subsequent work.

3.2. Choosing the Threshold

The selection of a suitable threshold u is a difficult problem that has been well-studied in the EVT literature. For a survey of approaches for setting the threshold, see Scarrott and MacDonald (2012). Many such approaches involve applying judgment to ultimately select a value of u. Typically, sensitivity analyses are performed by altering the threshold values and ensuring results are robust to the choice of u. However, a challenging aspect of threshold selection in the machine learning context of the current paper is that u must be decided automatically. We apply the recently developed method of Bader et al. (2018), which uses a combination of ordered goodness-of-fits tests and a stopping rule to choose the optimal threshold. This method provides some assurance that the excesses above the chosen threshold are sufficiently well-approximated the GPD.

The method of Bader et al. (2018) is as follows. Consider a fixed set of thresholds $u_1<\dots <u_l$, where, for each $u_i$, we have $k_i$ excesses. The sequence of null hypotheses for each respective test i, $i=1,\dots ,l$, is given by

$H_0^{\left(i\right)}:$	The distribution of the $k_i$ excesses above $u_i$
	follows the GPD.

For each threshold $u_i$, let ${\widehat{\theta }}_i=({\widehat{\xi }}_{u_i}^{\left(n\right)},{\widehat{\sigma }}_{u_i}^{\left(n\right)})$ denote the MLEs computed from the $k_i$ excesses above $u_i$. The Anderson–Darling (AD) test statistic comparing the empirical threshold excesses distribution with the GPD is then calculated. Let $y_{\left(1\right)}<\dots <y_{\left(k_i\right)}$ denote the ordered threshold excesses for test i and apply the transformation $z_{\left(j\right)}=G_{{\widehat{\theta }}_i}\left(y_{\left(j\right)}\right)$, $j=1,\dots k_i$, where G is from Equation (6). The AD statistic for test i is then

$$A_i^2=-k_i-\frac{1}{k_i}\sum _{j=1}^{k_i}(2j-1)\left[\log \left(z_{\left(j\right)}\right)+\log \left(1-z_{(k_i+1-j)}\right)\right].$$

(12)

Corresponding p-values for each test statistic can then be found by referring to a lookup table (e.g., Choulakian and Stephens 2001) or computed on-the-fly. In the present work, such p-values are obtained with the eva package in R. Using the p-values $p_1,\dots ,p_l$ calculated for each test, the ForwardStop rule of G’Sell et al. (2016) was used to choose the threshold. This is done by calculating

$${\widehat{w}}_F=max\left\{w\in I|-\frac{1}{w}\sum _{i=1}^w\log \left(1-p_i\right)\le \gamma \right\},$$

(13)

where $\gamma$ is a chosen significance parameter and $I\subseteq \{1,\dots ,l\}$, $I\ne \varnothing$. Under this rule, the threshold $u_v$ is chosen, where $v=min\{w\in I|w>{\widehat{w}}_F\}$. If ${\widehat{w}}_F$ does exist, then no rejection is made and $u_{min\left(I\right)}$ is chosen. If ${\widehat{w}}_F=max\left(I\right)$, then $u_{max\left(I\right)}$ is chosen. The overall CVaR estimation procedure is summarized in Algorithm 1.

Remark 1.

In the threshold selection procedure of Bader et al. (2018), ${\widehat{w}}_F$ is given with $I=\{1,\dots ,l\}$; however, we make the modification that I is an arbitrary index set in view of CVaR estimation: since ${\widehat{c}}_{u,\alpha }^{\left(n\right)}$ tends to infinity when ${\widehat{\xi }}_u^{\left(n\right)}$ tends to 1, in order to ensure reasonable estimates of the CVaR, we use a cutoff parameter ${\xi }_{max}<1$, where the MLE ${\widehat{\xi }}_{u_i}^{\left(n\right)}$ and corresponding threshold $u_i$ are discarded if ${\widehat{\xi }}_{u_i}^{\left(n\right)}>{\xi }_{max}$.

Remark 2.

Instead of choosing the candidate thresholds $u_1,\dots ,u_l$ directly, it is usually more convenient to choose threshold percentiles $q_1,\dots ,q_l$ and compute u values via the empirical quantile function, i.e., $u_i={\widehat{F}}_n^{-1}\left(q_i\right)$.

Remark 3.

It may happen that Algorithm 1 fails if $I=\varnothing$, in which case no suitable estimates of ξ are found. This is an indication that the underlying data distribution does not satisfy the condition $\xi <1$ and that the CVaR does not exist. To make Algorithm 1 robust, the sample average estimate is used as a fallback when the latter occurs.

Algorithm 1: EVT CVaR estimation with automated threshold selection

In the next section, we combine Algorithm 1 with an algorithm for best-arm identification in the risk-averse MAB framework.

4. Algorithm for Risk-Averse MABs

The MAB framework of this paper involves a finite horizon multi-stage decision setting, where an agent makes decisions at stages $t=1,\dots ,n$. Let $\{1,\dots ,K\}$ denote a set of arms, which are possible actions that can be taken at each stage. In the risk-averse setting, we consider the outcome of each draw from an arm to be a cost to the agent. Define the K-dimensional random vector $X^t\equiv (X_1^t,\dots ,X_K^t)$, where $X_j^t$ denotes the cost incurred if the arm j is selected at stage t. Vectors $X^1,\dots ,X^n$ are assumed to be independent and identically distributed.

Therefore, for all arms $i=1,\dots ,K$, cost variables $X_i^1,\dots ,X_i^n$ are i.i.d. copies of some random variable $X_i$. Let $\{F_1,\dots ,F_K\}$ denote the respective CDFs of $X_1,\dots ,X_K$; these distribution functions are unknown to the agent. Throughout the rest of the paper, it is assumed that each arm’s cost distribution satisfies Assumption 1. The integrability assumption that ${\xi }_i<1$, $i=1,\dots ,K$ is equivalent to the bounded moment condition often seen when analyzing bandit algorithms under risk criteria in the heavy-tailed domain (e.g., Bhat and Prashanth 2019), which is required for the CVaR to be finite.

We consider the problem of identifying the arm with the lowest CVaR value (at a pre-specified value of $\alpha$) in a fixed budget of n stages, where the CVaR is measured on single stage losses instead of on an aggregate of rewards across various stages. The sequence of selected arms is denoted by $a\equiv (a_1,\dots ,a_n)$, where $a_t\in \{1,\dots ,K\}$. When an arm $a_t$ is selected at stage t, its associated cost $X_{a_t}^t$ is observed; however, the costs associated with all other arms $\left\{X_i^t\right|i\ne a_t\}$ remain unobserved. Let $i^{*}={argmin}_{i=1,\dots ,K}CVa{R}_{\alpha }\left(X_i\right)$, i.e., the arm with the lowest CVaR value. At the end of n stages, the bandit algorithm outputs an arm selection ${\widehat{i}}^{*}\in \{1,\dots ,K\}$, which is perceived to be optimal by the learning agent.

The best-arm identification algorithm we consider is based on the successive rejects (SR) algorithm of Audibert and Bubeck (2010), modified to select the arm with lowest CVaR value as in Prashanth et al. (2020). The novel aspect of the proposed algorithm is it inclusion of the EVT-based estimation of CVaR. The steps of our original algorithm are presented in Algorithm 2. The algorithm proceeds through $K-1$ phases, eliminating the arm with the highest estimated CVaR value at each phase. The number of samples taken from each arm at each phase is designed such that the total arm samples does not exceed the given fixed budget n at the end of $K-1$ phases. For arms $i=1,\dots ,K$, we denote CVaR estimates using Algorithm 1 by ${\widehat{c}}_{u,\alpha }^{\left(m\right)}\left(i\right)$ (at a sample of size m).

Algorithm 2: EVT CVaR-SR algorithm

5. Numerical Experiments

In this section, two estimation methods for the CVaR are compared within a simulation study: the sample average estimator of Equation (3) and the EVT estimator of Algorithm 1, which we denote in this section by CVaR-SA and CVaR-EVT, respectively. First, we consider a pure statistical estimation problem where i.i.d. costs from a single arm are sequentially observed, and the cost distribution CVaR estimates are updated at every stage. This allows evaluating the statistical accuracy of both methods. The second experiment embeds the two respective CVaR estimation methods within the MAB problem of Algorithm 2 so as to assess their suitability for sequential action selection. All code to reproduce our results is provided on the following GitHub repository: https://github.com/dtroop/evt-bandits, accessed on 1 April 2022.

5.1. Single-Arm CVaR Estimation Experiment

Distributional parameters as in Section The single-arm problem where all costs are i.i.d. samples from an unknown distribution is first considered. The estimation performance of CVaR-SA and CVaR-EVT are compared via the root-mean-square error (RMSE) and absolute bias on five examples from each of the Burr, Fréchet, half-t, Lognormal, and Weibull class of distributions.

The Burr, Fréchet, and half-t distributions are all truly heavy-tailed in the sense that they satisfy Assumption 1 with $\xi >0$. The Lognormal and Weibull distributions can be considered borderline between light and heavy-tailed in the sense that they satisfy Assumption 1 with $\xi =0$; however, their tails can be heavier than exponential depending on their parameters. The distribution function and CVaR formula for each class of distribution is given in the Appendix A.

The experiments consist of performing $N=1000$ independent runs. Each run consists of sequentially sampling a total of $n=20,000$ independent costs from the single arm, and the CVaR estimates are updated at every interval of 2000 costs. We fix $\alpha =0.998$ as an example of an extreme risk level. To compute CVaR-EVT, we set $\gamma =0.1$ and ${\xi }_{max}=0.9$ in Algorithm 1, with 20 threshold percentiles $q_1=0.79,q_2=0.80,\dots ,q_{20}=0.98$. Plots of the RMSE and the absolute bias from estimating the CVaR across independent runs on simulated data are given in Figure 1 and Figure 2, respectively. Table A1 of Appendix B provides the mean CVaR estimates across runs at the last stage and corresponding true CVaR values. The latter table also contains information related to the percentage of thresholds rejected in the simulations due to ${\widehat{\xi }}_{u_i}^{\left(n\right)}>{\xi }_{max}$, and the average threshold quantile $q_i$ selected in the simulations for each considered distribution.

Figure 1. RMSE of CVaR-EVT and CVaR-SA with $\alpha =0.998$.

Figure 2. Absolute bias of CVaR-EVT and CVaR-SA with $\alpha =0.998$.

In terms of RMSE, the performance improvement of CVaR-EVT over CVaR-SA is most apparent in the Burr, Fréchet, and half-t distributions, where $\xi >0$. These distributions have particularly long tails, and so the few samples used in CVaR-SA do not adequately capture a good estimate of the CVaR. The average threshold percentile chosen by Algorithm 1 for these distributions at $n=20,000$ ranges from $0.80$ to $0.82$, which highlights the fact that CVaR-EVT makes use of a significant portion of data below the $0.998$ quantile.

The Lognormal and Weibull distributions have tails that are more distinct from the GPD, and while we still notice improved overall accuracy from using the additional samples in CVaR-EVT, this improvement is less significant than in the truly heavy-tailed cases. CVaR-SA actually outperforms CVaR-EVT in some cases, which occurs most significantly in the Lognormal(1, 1.5) and Weibull(0.5, 1) distributions. The average threshold percentile chosen in these distributions at $n=20,000$ was $0.84$ and $0.90$, respectively, indicating that the GPD model did not fit well unless a higher threshold is used in CVaR-EVT.

In Table A1, we provide the average threshold selected by Algorithm 1 for all distributions, as well as the average number of thresholds rejected by the cutoff rule (i.e., where ${\widehat{\xi }}_{u_i}^{\left(n\right)}>{\xi }_{max}$). The latter is reported at a sample size of $n=2000$ since extremely few thresholds were rejected beyond this sample size.

Despite often exhibiting better performance in terms of RMSE, the CVaR-EVT leads to higher bias than the CVaR-SA for several of the simulations, see for instance the two rightmost columns of Figure 2. This is unsurprising since, as mentioned earlier, estimates of the GPD parameters with the maximum likelihood from Equation (9) are known to be asymptotically biased (De Haan and Ferreira 2006). The use of de-biasing procedures for the CVaR-EVT estimates, such as that of Troop et al. (2021) could therefore be contemplated to further improve the estimation performance.

5.2. Multi-Armed Bandit Experiment

In this section, we compare the performance of CVaR-EVT and CVaR-SA in a best-arm identification problem. Algorithm 2 is compared with Prashanth et al. (2020, Algorithm 1) (i.e., where CVaR estimates are computed using CVaR-SA) in five five-arm MAB settings, where in each setting, the five underlying arm distributions come from the same distribution class, either the Burr, Fréchet, half-t, Lognormal or Weibull. Again, we fix $\alpha =0.998$ and use the same distributional parameters as in Section 5.1. In other words, each of the five arms in the bandits experiment has one of the five parameter sets presented in the first experiment. Such parameters were chosen to vary the shape of the underlying distributions while keeping the arm CVaRs relatively close in magnitude, making the task of distinguishing them more difficult to the agent. Both Algorithm 2 and Prashanth et al. (2020, Algorithm 1) are executed at fixed budgets of n = 5000, 10,000, …, 25,000, and $N=1000$ independent runs are performed at each budget. The performance metric we use to compare the two methods is the probability of incorrect best-arm identification, i.e., $\mathbb{P}({\widehat{i}}^{*}\ne i^{*})$, estimated empirically at a given fixed budget by

$$\frac{{\sum }_{j=1}^N𝟙\{{\widehat{i}}_j^{*}\ne i^{*}\}}{N},$$

where ${\widehat{i}}_j^{*}$ denotes the estimated optimal arm at the considered stage for independent run j, $j=1,\dots ,N$, and $i^{*}$ denotes the arm with the lowest CVaR value. Plots of the empirical estimates of the probability of incorrect best-arm identification using CVaR-EVT and CVaR-SA are given in Figure 3. Consistently with the results of Section 5.1, the improved accuracy of CVaR-EVT in these classes of distributions leads to a lower error rate in the MAB setting for all budgets in our experiments compared to CVaR-SA.

Figure 3. The empirical probability of incorrect best-arm identification using CVaR-EVT and CVaR-SA in the successive rejects algorithm.

6. Conclusions

We developed a new algorithm for CVaR estimation combining EVT and automated threshold selection and used it in a risk-averse, best-arm identification MAB problem. While the CVaR approximation of Equation (8) exists in the literature, its efficacy in statistical estimation is limited by the problem of threshold selection, which can be unreliable in practice.

The novelty of our approach from a computational perspective is to integrate the sequential goodness-of-fit test of Bader et al. (2018) to develop an automated CVaR estimation algorithm using EVT in view of sequential decision making based on extreme risk. We demonstrated empirically that our algorithm produced more accurate estimates compared to the more commonly encountered sample average CVaR estimator in certain distributions, and is a preferable choice in the risk-averse MAB setting when the underlying arm distributions are heavy-tailed and the confidence level of the CVaR $\alpha$ is large, i.e., close to one.

The present paper has implications from both the academic and the practitioners’ perspective. First, our work is a useful addition to the risk-aware reinforcement learning strand of literature, where the key differentiating feature of the present paper is the focus on extreme risk. Moreover, our results also have implications for practitioners as our algorithm can readily be deployed by risk managers to optimize the exposure to catastrophic risk in the context of sequential decision making, for instance in financial contexts.

Among future potential work, testing the proposed algorithm on real data would be of interest. Indeed, while the present study establishes that our algorithm provides good performance in a controlled environment with simulated heavy-tailed data, confirming that such favorable performance still persists in a realistic context is important. Furthermore, as a potential extension, the de-biasing procedure for EVT CVaR estimates from Troop et al. (2021) could be integrated into the multi-armed bandits selection algorithm to further improve the results. Finally, considering alternative objective functions instead of the CVaR could also be explored. For instance, despite being harder to interpret, expectiles are a popular choice in the risk management literature due to their favorable properties, such as elicitability; see for instance Daouia et al. (2018). The extreme-value-based estimation of expectiles could thus also be integrated into a risk-aware MAB framework.

Appendix A.1. Burr

The Burr distribution with parameters $c,d$ has CDF given by

$$F_{c,d}\left(x\right)=1-{\left(1+x^c\right)}^{-d},c,d,x>0.$$

The CVaR for the Burr distribution can be derived from its expression for the conditional moment given in Kumar (2017, Section 2.2). If $X\sim Burr(c,d)$,

$${\mathrm{CVaR}}_{\alpha }\left(X\right)=\frac{d{\left[{(1/q_{\alpha })}^c\right]}^{d-1/c}}{(1-\alpha )(d-1/c)}{}_2{F}_1\left(d-\frac{1}{c},1+d,d-\frac{1}{c}+1;-\frac{1}{q_{\alpha }}\right),cd>1,$$

(A1)

where ${}_2{F}_1$ denotes the hypergeometric function and $q_{\alpha }=F_{c,d}^{-1}\left(\alpha \right)$. The Burr distribution is in $\mathrm{MDA}\left(H_{\xi }\right)$ with $\xi =1/\left(cd\right)$.

Appendix A.2. Fréchet

The Fréchet distribution with parameter $\gamma$ has CDF given by

$$F_{\gamma }\left(x\right)=e^{-x^{-\gamma }},\gamma ,x>0.$$

The CVaR for the Fréchet distribution can be derived from Equation (2). If $X\sim Fréchet\left(\gamma \right)$,

$${\mathrm{CVaR}}_{\alpha }\left(X\right)={(1-\alpha )}^{-1}\left[\Gamma \left(\gamma -1/\gamma \right)\right.-\left.\Gamma \left(\gamma -1/\gamma ,-\log \left(\alpha \right)\right)\right],\gamma >1,$$

(A2)

where $\Gamma (·)$ and $\Gamma (·,·)$ denote the gamma and upper incomplete gamma functions, respectively. The Fréchet distribution is in $\mathrm{MDA}\left(H_{\xi }\right)$ with $\xi =1/\gamma$.

Appendix A.3. Half-t

If X follows the t distribution with $\nu$ degrees of freedom, then $\left|X\right|$ follows the half-t distribution, which has a CDF given by

$$F_{\nu }\left(x\right)=2-{\mathcal{I}}_{t\left(x\right)}\left(\frac{\nu }{2},\frac{1}{2}\right),\nu >0,x\ge 0,$$

where $t\left(x\right)=\frac{\nu }{x^2+\nu }$ and ${\mathcal{I}}_t(a,b)$ is the regularized incomplete Beta function. The CVaR for the half-t distribution can be derived from the expression for the CVaR of the t-distribution given in (Norton et al. 2019, Proposition 12). If $X\sim half-t\left(\nu \right)$, then

$${\mathrm{CVaR}}_{\alpha }\left(X\right)=2\frac{\nu +q_{\alpha }}{(\nu -1)(1-\alpha )}{g}_{\nu }\left(q_{\alpha }\right),\nu >1,$$

where $g_{\nu }$ is the probability density function of the standardized t-distribution, and $q_{\alpha }=T^{-1}\left(\frac{\alpha +1}{2}\right)$ where $T^{-1}$ is the inverse of the CDF of standardized t-distribution. The half-t distribution is in $\mathrm{MDA}\left(H_{\xi }\right)$ with $\xi =1/\nu$.

Appendix A.4. Lognormal

The Lognormal distribution with parameters $\mu ,\sigma$ has a CDF given by

$$F_{\mu ,\sigma }\left(x\right)=\frac{1}{2}+\frac{1}{2}erf\left[\frac{lnx-\mu }{\sqrt{2}\sigma }\right],\sigma ,x>0.$$

The CVaR for the Lognormal distribution is given in (Norton et al. 2019, Proposition 9). If $X\sim Lognormal(\mu ,\sigma )$,

$${\mathrm{CVaR}}_{\alpha }\left(X\right)=\frac{e^{\mu +{\sigma }^2/2}}{1-\alpha }\Phi \left[\sigma -\frac{{\Phi }^{-1}\left(\alpha \right)}{\sqrt{2}}\right],$$

where $\Phi$ and ${\Phi }^{-1}$ are, respectively, the standard normal CDF and its inverse. The Lognormal distribution is in $\mathrm{MDA}\left(H_{\xi }\right)$ with $\xi =0$.

Appendix A.5. Weibull

The Weibull distribution with parameters $\kappa ,\lambda$ has a CDF given by

$$F_{\kappa ,\lambda }\left(x\right)=1-e^{-{(x/\lambda )}^{\kappa }},\kappa ,\lambda >0,x\ge 0.$$

The CVaR for the Weibull distribution is given in (Norton et al. 2019, Proposition 13). If $X\sim \mathrm{Weibull}(\kappa ,\lambda )$,

$${\mathrm{CVaR}}_{\alpha }\left(X\right)=\frac{\lambda }{1-\alpha }\Gamma \left(1+\frac{1}{\kappa },-\log (1-\alpha )\right),$$

where $\Gamma (a,b)={\int }_b^{\infty }{p}^{a-1}{e}^{-p}dp$ is the upper incomplete gamma function. The Weibull distribution is in $\mathrm{MDA}\left(H_{\xi }\right)$ with $\xi =0$.