The Annals of Applied Probability

Randomized urn models revisited using stochastic approximation

Sophie Laruelle and Gilles Pagès

Full-text: Open access


This paper presents the link between stochastic approximation and clinical trials based on randomized urn models investigated by Bai and Hu [Stochastic Process. Appl. 80 (1999) 87–101], Bai and Hu [Ann. Appl. Probab. 15 (2005) 914–940] and Bai, Hu and Shen [J. Multivariate Anal. 81 (2002) 1–18]. We reformulate the dynamics of both the urn composition and the assigned treatments as standard stochastic approximation (SA) algorithms with remainder. Then, we derive the a.s. convergence and the asymptotic normality [central limit theorem (CLT)] of the normalized procedure under less stringent assumptions by calling upon the ODE and SDE methods. As a second step, we investigate a more involved family of models, known as multi-arm clinical trials, where the urn updating depends on the past performances of the treatments. By increasing the dimension of the state vector, our SA approach provides this time a new asymptotic normality result.

Article information

Ann. Appl. Probab., Volume 23, Number 4 (2013), 1409-1436.

First available in Project Euclid: 21 June 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62L20: Stochastic approximation 62E20: Asymptotic distribution theory 62L05: Sequential design
Secondary: 62F12: Asymptotic properties of estimators 62P10: Applications to biology and medical sciences

Stochastic approximation extended Pólya urn models nonhomogeneous generating matrix strong consistency asymptotic normality multi-arm clinical trials adaptive asset allocation


Laruelle, Sophie; Pagès, Gilles. Randomized urn models revisited using stochastic approximation. Ann. Appl. Probab. 23 (2013), no. 4, 1409--1436. doi:10.1214/12-AAP875.

Export citation


  • [1] Andersen, J., Faries, D. and Tamura, R. (1994). A randomized play-the-winner design for multi-arm clinical trials. Comm. Statist. Theory Methods 23 309–323.
  • [2] Athreya, K. B. and Karlin, S. (1967). Limit theorems for the split times of branching processes. J. Math. Mech. 17 257–277.
  • [3] Athreya, K. B. and Karlin, S. (1968). Embedding of urn schemes into continuous time Markov branching processes and related limit theorems. Ann. Math. Statist. 39 1801–1817.
  • [4] Bagchi, A. and Pal, A. K. (1985). Asymptotic normality in the generalized Pólya–Eggenberger urn model, with an application to computer data structures. SIAM J. Algebraic Discrete Methods 6 394–405.
  • [5] Bai, Z. D. and Hu, F. (1999). Asymptotic theorems for urn models with nonhomogeneous generating matrices. Stochastic Process. Appl. 80 87–101.
  • [6] Bai, Z.-D. and Hu, F. (2005). Asymptotics in randomized URN models. Ann. Appl. Probab. 15 914–940.
  • [7] Bai, Z. D., Hu, F. and Shen, L. (2002). An adaptive design for multi-arm clinical trials. J. Multivariate Anal. 81 1–18.
  • [8] Benaïm, M. (1999). Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII. Lecture Notes in Math. 1709 1–68. Springer, Berlin.
  • [9] Benveniste, A., Métivier, M. and Priouret, P. (1990). Adaptive Algorithms and Stochastic Approximations. Applications of Mathematics (New York) 22. Springer, Berlin. Translated from the French by Stephen S. Wilson.
  • [10] Berman, A. and Plemmons, R. J. (1994). Nonnegative Matrices in the Mathematical Sciences. Classics in Applied Mathematics 9. SIAM, Philadelphia, PA. Revised reprint of the 1979 original.
  • [11] Bouton, C. (1988). Approximation gaussienne d’algorithmes stochastiques à dynamique markovienne. Ann. Inst. Henri Poincaré Probab. Stat. 24 131–155.
  • [12] Cheung, Y. K. (2010). Stochastic approximation and modern model-based designs for dose-finding clinical trials. Statist. Sci. 25 191–201.
  • [13] Duflo, M. (1996). Algorithmes Stochastiques. Mathématiques & Applications (Berlin) [Mathematics & Applications] 23. Springer, Berlin.
  • [14] Duflo, M. (1997). Random Iterative Models. Applications of Mathematics (New York) 34. Springer, Berlin. Translated from the 1990 French original by Stephen S. Wilson and revised by the author.
  • [15] Flournoy, N. and Rosenberger, W. F., eds. (1995). Adaptive Designs. Institute of Mathematical Statistics Lecture Notes—Monograph Series 25. IMS, Hayward, CA.
  • [16] Fort, J.-C. and Pagès, G. (1996). Convergence of stochastic algorithms: From the Kushner–Clark theorem to the Lyapounov functional method. Adv. in Appl. Probab. 28 1072–1094.
  • [17] Freedman, D. A. (1965). Bernard Friedman’s urn. Ann. Math. Statist. 36 956–970.
  • [18] Gouet, R. (1993). Martingale functional central limit theorems for a generalized Pólya urn. Ann. Probab. 21 1624–1639.
  • [19] Janson, S. (2004). Functional limit theorems for multitype branching processes and generalized Pólya urns. Stochastic Process. Appl. 110 177–245.
  • [20] Johnson, N. L. and Kotz, S. (1977). Urn Models and Their Application: An Approach to Modern Discrete Probability Theory. Wiley, New York.
  • [21] Kushner, H. J. and Clark, D. S. (1978). Stochastic Approximation Methods for Constrained and Unconstrained Systems. Applied Mathematical Sciences 26. Springer, New York.
  • [22] Kushner, H. J. and Yin, G. G. (2003). Stochastic Approximation and Recursive Algorithms and Applications: Stochastic Modelling and Applied Probability, 2nd ed. Applications of Mathematics (New York) 35. Springer, New York.
  • [23] Lamberton, D., Pagès, G. and Tarrès, P. (2004). When can the two-armed bandit algorithm be trusted? Ann. Appl. Probab. 14 1424–1454.
  • [24] Pouyanne, N. (2008). An algebraic approach to Pólya processes. Ann. Inst. Henri Poincaré Probab. Stat. 44 293–323.
  • [25] Rosenberger, W. F. (1996). New directions in adaptive designs. Statist. Sci. 11 137–149.
  • [26] Smythe, R. T. (1996). Central limit theorems for urn models. Stochastic Process. Appl. 65 115–137.
  • [27] Wei, L. J. (1979). The generalized Polya’s urn design for sequential medical trials. Ann. Statist. 7 291–296.
  • [28] Zelen, M. (1969). Play the winner rule and the controlled clinical trial. J. Amer. Statist. Assoc. 64 131–146.