August 2022 Sequential estimation of quantiles with applications to A/B testing and best-arm identification
Steven R. Howard, Aaditya Ramdas
Author Affiliations +
Bernoulli 28(3): 1704-1728 (August 2022). DOI: 10.3150/21-BEJ1388


We design confidence sequences—sequences of confidence intervals which are valid uniformly over time—for quantiles of any distribution over a complete, fully-ordered set, based on a stream of i.i.d. observations. We give methods both for tracking a fixed quantile and for tracking all quantiles simultaneously. Specifically, we provide explicit expressions with small constants for intervals whose widths shrink at the fastest possible t1loglogt rate, along with a nonasymptotic concentration inequality for the empirical distribution function which holds uniformly over time with the same rate. The latter strengthens Smirnov’s empirical process law of the iterated logarithm and extends the Dvoretzky-Kiefer-Wolfowitz inequality to hold uniformly over time. We give a new algorithm and sample complexity bound for selecting an arm with an approximately best quantile in a multi-armed bandit framework. In simulations, our method needs fewer samples than existing methods by a factor of five to fifty.

Funding Statement

Howard thanks Office of Naval Research (ONR) Grant N00014-15-1-2367.


We thank Jon McAuliffe for helpful comments.


Download Citation

Steven R. Howard. Aaditya Ramdas. "Sequential estimation of quantiles with applications to A/B testing and best-arm identification." Bernoulli 28 (3) 1704 - 1728, August 2022.


Received: 1 August 2019; Published: August 2022
First available in Project Euclid: 25 April 2022

MathSciNet: MR4411508
zbMATH: 07526603
Digital Object Identifier: 10.3150/21-BEJ1388

Keywords: best-arm identification , Confidence sequences , Dvoretzky-Kiefer-Wolfowitz inequality , empirical process , Quantile estimation


This article is only available to subscribers.
It is not available for individual sale.

Vol.28 • No. 3 • August 2022
Back to Top