Open Access
June 2013 Kullback–Leibler upper confidence bounds for optimal sequential allocation
Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, Gilles Stoltz
Ann. Statist. 41(3): 1516-1541 (June 2013). DOI: 10.1214/13-AOS1119

Abstract

We consider optimal sequential allocation in the context of the so-called stochastic multi-armed bandit model. We describe a generic index policy, in the sense of Gittins [J. R. Stat. Soc. Ser. B Stat. Methodol. 41 (1979) 148–177], based on upper confidence bounds of the arm payoffs computed using the Kullback–Leibler divergence. We consider two classes of distributions for which instances of this general idea are analyzed: the kl-UCB algorithm is designed for one-parameter exponential families and the empirical KL-UCB algorithm for bounded and finitely supported distributions. Our main contribution is a unified finite-time analysis of the regret of these algorithms that asymptotically matches the lower bounds of Lai and Robbins [Adv. in Appl. Math. 6 (1985) 4–22] and Burnetas and Katehakis [Adv. in Appl. Math. 17 (1996) 122–142], respectively. We also investigate the behavior of these algorithms when used with general bounded rewards, showing in particular that they provide significant improvements over the state-of-the-art.

Citation

Download Citation

Olivier Cappé. Aurélien Garivier. Odalric-Ambrym Maillard. Rémi Munos. Gilles Stoltz. "Kullback–Leibler upper confidence bounds for optimal sequential allocation." Ann. Statist. 41 (3) 1516 - 1541, June 2013. https://doi.org/10.1214/13-AOS1119

Information

Published: June 2013
First available in Project Euclid: 1 August 2013

zbMATH: 1293.62161
MathSciNet: MR3113820
Digital Object Identifier: 10.1214/13-AOS1119

Subjects:
Primary: 62L10 , 62L12 , 68T05

Keywords: Kullback–Leibler divergence , Multi-armed bandit problems , sequential testing , upper confidence bound

Rights: Copyright © 2013 Institute of Mathematical Statistics

Vol.41 • No. 3 • June 2013
Back to Top