Bayesian Nonparametric Bandits

Abstract

Sequential selections are to be made from two stochastic processes, or "arms." At each stage the arm selected for observation depends on past observations. The objective is to maximize the expected sum of the first $n$ observations. For arm 1 the observations are identically distributed with probability measure $P$, and for arm 2 the observations have probability measure $Q; P$ is a Dirichlet process and $Q$ is known. An equivalent problem is deciding sequentially when to stop sampling from an unknown population. Optimal strategies are shown to continue sampling if the current observation is sufficiently large. A simple form of such a rule is expressed in terms of a degenerate Dirichlet process which is related to $P$.