Electronic Journal of Probability

Nonlinear randomized urn models: a stochastic approximation viewpoint

Sophie Laruelle and Gilles Pagès

Full-text: Open access


This paper extends the link between stochastic approximation ($SA$) theory and randomized urn models developed in [32], and their applications to clinical trials introduced in [2, 3, 4]. We no longer assume that the drawing rule is uniform among the balls of the urn (which contains $d$ colors), but can be reinforced by a function $f$. This is a way to model risk aversion. Firstly, by considering that $f$ is concave or convex and by reformulating the dynamics of the urn composition as an $SA$ algorithm with remainder, we derive the $a.s.$ convergence and the asymptotic normality (Central Limit Theorem, $CLT$) of the normalized procedure by calling upon the so-called $ODE$ and $SDE$ methods. An in depth analysis of the case $d=2$ exhibits two different behaviors: a single equilibrium point when $f$ is concave, and, when $f$ is convex, a transition phase from a single attracting equilibrium to a system with two attracting and one repulsive equilibrium points. The last setting is solved using results on non-convergence toward noisy and noiseless “traps” in order to deduce the $a.s.$ convergence toward one of the attracting points. Secondly, the special case of a Pólya urn (when the addition rule is the $I_{d}$ matrix) is analyzed, still using result from $SA$ theory about “traps”. Finally, these results are used to solve another urn model with a more natural nonlinear drawing rule and we conclude by an example of application to optimal asset allocation in Finance.

Article information

Electron. J. Probab., Volume 24 (2019), paper no. 98, 47 pp.

Received: 23 May 2018
Accepted: 29 April 2019
First available in Project Euclid: 18 September 2019

Permanent link to this document

Digital Object Identifier

Primary: 62L20: Stochastic approximation 62E20: Asymptotic distribution theory 62L05: Sequential design
Secondary: 62F12: Asymptotic properties of estimators 62P10: Applications to biology and medical sciences

stochastic approximation extended Pólya urn models reinforcement non-homogeneous generating matrix strong consistency asymptotic normality bandit algorithms

Creative Commons Attribution 4.0 International License.


Laruelle, Sophie; Pagès, Gilles. Nonlinear randomized urn models: a stochastic approximation viewpoint. Electron. J. Probab. 24 (2019), paper no. 98, 47 pp. doi:10.1214/19-EJP312. https://projecteuclid.org/euclid.ejp/1568793792

Export citation


  • [1] K. B. Athreya and S. Karlin. Embedding of urn schemes into continuous time Markov branching processes and related limit theorems. Ann. Math. Statist., 39:1801–1817, 1968.
  • [2] Z.-D. Bai and F. Hu. Asymptotic theorems for urn models with nonhomogeneous generating matrices. Stochastic Process. Appl., 80(1):87–101, 1999.
  • [3] Z.-D. Bai and F. Hu. Asymptotics in randomized urn models. Ann. Appl. Probab., 15(1B):914–940, 2005.
  • [4] Z.-D. Bai, F. Hu, and L. Shen. An adaptive design for multi-arm clinical trials. J. Multivariate Anal., 81(1):1–18, 2002.
  • [5] M. Benaïm. Recursive algorithms, urn processes and chaining number of chain recurrent sets. Ergodic Theory Dynam. Systems, 18(1):53–87, 1998.
  • [6] M. Benaïm. Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII, volume 1709 of Lecture Notes in Math., pages 1–68. Springer, Berlin, 1999.
  • [7] M. Benaïm, I. Benjamini, J. Chen, and Y. Lima. A generalized Pólya’s urn with graph based interactions. Random Structures Algorithms, 46(4):614–634, 2015.
  • [8] M. Benaïm and M. W. Hirsch. Asymptotic pseudotrajectories and chain recurrent flows, with applications. J. Dynam. Differential Equations, 8(1):141–176, 1996.
  • [9] A. Benveniste, M. Métivier, and P. Priouret. Adaptive algorithms and stochastic approximations, volume 22 of Applications of Mathematics (New York). Springer-Verlag, Berlin, 1990. Translated from the French by Stephen S. Wilson.
  • [10] N.H. Bingham, C.M. Goldie, and J.L. Teugels. Regular Variation. Encyclopedia of Mathematics and its Applications. Cambridge University Press, 1989.
  • [11] C. Bouton. Approximation gaussienne d’algorithmes stochastiques à dynamique markovienne. Ann. Inst. H. Poincaré Probab. Statist., 24(1):131–155, 1988.
  • [12] O. Brandière and M. Duflo. Les algorithmes stochastiques contournent-ils les pièges? Ann. Inst. H. Poincaré Probab. Statist., 32(3):395–427, 1996.
  • [13] B. Chauvin, C. Mailler, and N. Pouyanne. Smoothing equations for large Pólya urns. J. Theoret. Probab., 28(3):923–957, 2015.
  • [14] B. Chauvin, N. Pouyanne, and R. Sahnoun. Limit distributions for large polya urns. Ann. Appl. Probab., 21(1):1–32, 2011.
  • [15] J. Chen and C. Lucas. A generalized Pólya’s urn with graph based interactions: convergence at linearity. Electron. Commun. Probab., 19:no. 67, 13, 2014.
  • [16] A. Collevecchio, C. Cotar, and M. LiCalzi. On a preferential attachment and generalized Pòlya’s urn model. Ann. Appl. Probab., 23(3):1219–1253, 2013.
  • [17] M. Duflo. Algorithmes stochastiques, volume 23 of Mathématiques & Applications (Berlin) [Mathematics & Applications]. Springer-Verlag, Berlin, 1996.
  • [18] M. Duflo. Random iterative models, volume 34 of Applications of Mathematics (New York). Springer-Verlag, Berlin, 1997. Translated from the 1990 French original by Stephen S. Wilson and revised by the author.
  • [19] J.-C. Fort and G. Pagès. Convergence of stochastic algorithms: from the Kushner-Clark theorem to the Lyapounov functional method. Adv. in Appl. Probab., 28(4):1072–1094, 1996.
  • [20] J.-C. Fort and G. Pagès. Decreasing step stochastic algorithms: a.s. behaviour of weighted empirical measures. Monte Carlo Methods Appl., 8(3):237–270, 2002.
  • [21] D. A. Freedman. Bernard Friedman’s urn. Ann. Math. Statist, 36:956–970, 1965.
  • [22] P. Hall and C. C. Heyde. Martingale limit theory and its application. Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London, 1980. Probability and Mathematical Statistics.
  • [23] E. Häusler and H. Luschgy. Stable convergence and stable limit theorems, volume 74 of Probability Theory and Stochastic Modelling. Springer, Cham, 2015.
  • [24] B. M. Hill, D. Lane, and W. Sudderth. Exchangeable urn processes. Ann. Probab., 15(4):1586–1592, 1987.
  • [25] S. Janson. Functional limit theorems for multitype branching processes and generalized Pólya urns. Stochastic Process. Appl., 110(2):177–245, 2004.
  • [26] M. Knape and R. Neininger. Pólya urns via the contraction method. Combin. Probab. Comput., 23(6):1148–1186, 2014.
  • [27] H. J. Kushner and D. S. Clark. Stochastic approximation methods for constrained and unconstrained systems, volume 26 of Applied Mathematical Sciences. Springer-Verlag, New York, 1978.
  • [28] H. J. Kushner and G. G. Yin. Stochastic approximation and recursive algorithms and applications, volume 35 of Applications of Mathematics (New York). Springer-Verlag, New York, second edition, 2003. Stochastic Modelling and Applied Probability.
  • [29] D. Lamberton and G. Pagès. How fast is the bandit? Stoch. Anal. Appl., 26(3):603–623, 2008.
  • [30] D. Lamberton and G. Pagès. A penalized bandit algorithm. Electron. J. Probab., 13:no. 13, 341–373, 2008.
  • [31] D. Lamberton, G. Pagès, and P. Tarrès. When can the two-armed bandit algorithm be trusted? Ann. Appl. Probab., 14(3):1424–1454, 2004.
  • [32] S. Laruelle and G. Pagès. Randomized urn models revisited using stochastic approximation. Ann. Appl. Probab., 23(4):1409–1436, 2013.
  • [33] S. Laruelle and G. Pagès. Addendum and corrigendum to “Randomized urn models revisited using stochastic approximation” []. Ann. Appl. Probab., 27(2):1296–1298, 2017.
  • [34] N. Lasmar, S. Mailler, and Selmi O. Multiple drawing multi-colour urns by stochastic approximation. J. of Appl. Probab., 55(1):254–281, 2018.
  • [35] V. A. Lazarev. Convergence of stochastic approximation procedures in the case of regression equation with several roots. Problemy Peredachi Informatsii, 28(1):75–88, 1992.
  • [36] L. Ljung. Analysis of recursive stochastic algorithms. IEEE Trans. Automatic Control, AC-22(4):551–575, 1977.
  • [37] R. Pemantle. Nonconvergence to unstable points in urn models and stochastic approximations. Ann. Probab., 18(2):698–712, 1990.
  • [38] N. Pouyanne. An algebraic approach to Pólya processes. Ann. Inst. Henri Poincaré, 44(2):293–323, 2008.
  • [39] R. van der Hofstad, M. Holmes, A. Kuznetsov, and W. Ruszel. Strongly reinforced Pólya urns with graph-based competition. Ann. Appl. Probab., 26(4):2494–2539, 2016.
  • [40] L.-X. Zhang. Central limit theorems of a recursive stochastic algorithm with applications to adaptive design. Ann. Appl. Prob., 26:3630–3658, 2016.
  • [41] T. Zhu. Nonlinear Polya urn models and self-organizing processes. ProQuest LLC, Ann Arbor, MI, 2009. Thesis (Ph.D.)–University of Pennsylvania.