Next Article in Journal
The Optimal Limit Prices of Limit Orders under an Extended Geometric Brownian Motion with Bankruptcy Risk
Next Article in Special Issue
A Bayesian Model of COVID-19 Cases Based on the Gompertz Curve
Previous Article in Journal
HTLV/HIV Dual Infection: Modeling and Analysis
Previous Article in Special Issue
On the Size of Subclasses of Quasi-Copulas and Their Dedekind–MacNeille Completion
Article

Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

Department of Statistics, Carlos III University of Madrid, 28903 Getafe, Spain
Mathematics 2021, 9(1), 52; https://doi.org/10.3390/math9010052
Received: 7 December 2020 / Revised: 23 December 2020 / Accepted: 24 December 2020 / Published: 29 December 2020
(This article belongs to the Special Issue Stochastic Models with Applications)
We consider the multi-armed bandit problem with penalties for switching that include setup delays and costs, extending the former results of the author for the special case with no switching delays. A priority index for projects with setup delays that characterizes, in part, optimal policies was introduced by Asawa and Teneketzis in 1996, yet without giving a means of computing it. We present a fast two-stage index computing method, which computes the continuation index (which applies when the project has been set up) in a first stage and certain extra quantities with cubic (arithmetic-operation) complexity in the number of project states and then computes the switching index (which applies when the project is not set up), in a second stage, with quadratic complexity. The approach is based on new methodological advances on restless bandit indexation, which are introduced and deployed herein, being motivated by the limitations of previous results, exploiting the fact that the aforementioned index is the Whittle index of the project in its restless reformulation. A numerical study demonstrates substantial runtime speed-ups of the new two-stage index algorithm versus a general one-stage Whittle index algorithm. The study further gives evidence that, in a multi-project setting, the index policy is consistently nearly optimal. View Full-Text
Keywords: multi-armed bandits; setup delays; setup costs; index policies; semi-Markov decision processes; hysteresis multi-armed bandits; setup delays; setup costs; index policies; semi-Markov decision processes; hysteresis
Show Figures

Figure 1

MDPI and ACS Style

Niño-Mora, J. Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays. Mathematics 2021, 9, 52. https://doi.org/10.3390/math9010052

AMA Style

Niño-Mora J. Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays. Mathematics. 2021; 9(1):52. https://doi.org/10.3390/math9010052

Chicago/Turabian Style

Niño-Mora, José. 2021. "Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays" Mathematics 9, no. 1: 52. https://doi.org/10.3390/math9010052

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop