The Annals of Applied Probability

Stochastic approximation with random step sizes and urn models with random replacement matrices having finite mean

Ujan Gangopadhyay and Krishanu Maulik

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


The stochastic approximation algorithm is a useful technique which has been exploited successfully in probability theory and statistics for a long time. The step sizes used in stochastic approximation are generally taken to be deterministic and same is true for the drift. However, the specific application of urn models with random replacement matrices motivates us to consider stochastic approximation in a setup where both the step sizes and the drift are random, but the sequence is uniformly bounded. The problem becomes interesting when the negligibility conditions on the errors hold only in probability. We first prove a result on stochastic approximation in this setup, which is new in the literature. Then, as an application, we study urn models with random replacement matrices.

In the urn model, the replacement matrices need neither be independent, nor identically distributed. We assume that the replacement matrices are only independent of the color drawn in the same round conditioned on the entire past. We relax the usual second moment assumption on the replacement matrices in the literature and require only first moment to be finite. We require the conditional expectation of the replacement matrix given the past to be close to an irreducible matrix, in an appropriate sense. We do not require any of the matrices to be balanced or nonrandom. We prove convergence of the proportion vector, the composition vector and the count vector in $L^{1}$, and hence in probability. It is to be noted that the related differential equation is of Lotka–Volterra type and can be analyzed directly.

Article information

Ann. Appl. Probab., Volume 29, Number 4 (2019), 2033-2066.

Received: August 2018
First available in Project Euclid: 23 July 2019

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62L20: Stochastic approximation
Secondary: 60F15: Strong theorems 60G42: Martingales with discrete parameter

Urn model random replacement matrix balanced replacement matrix irreducibility stochastic approximation random step size random drift uniform integrability Lotka–Volterra differential equation


Gangopadhyay, Ujan; Maulik, Krishanu. Stochastic approximation with random step sizes and urn models with random replacement matrices having finite mean. Ann. Appl. Probab. 29 (2019), no. 4, 2033--2066. doi:10.1214/18-AAP1441.

Export citation


  • Athreya, K. B. and Karlin, S. (1968). Embedding of urn schemes into continuous time Markov branching processes and related limit theorems. Ann. Math. Stat. 39 1801–1817.
  • Athreya, K. B. and Ney, P. E. (2004). Branching Processes. Dover, Mineola, NY.
  • Bai, Z.-D. and Hu, F. (2005). Asymptotics in randomized URN models. Ann. Appl. Probab. 15 914–940.
  • Benaïm, M. (1999). Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII. Lecture Notes in Math. 1709 1–68. Springer, Berlin.
  • Borkar, V. S. (2008). Stochastic Approximation. Cambridge Univ. Press, Cambridge.
  • Dasgupta, A. and Maulik, K. (2011). Strong laws for urn models with balanced replacement matrices. Electron. J. Probab. 16 1723–1749.
  • Freedman, D. A. (1965). Bernard Friedman’s urn. Ann. Math. Stat. 36 956–970.
  • Gouet, R. (1997). Strong convergence of proportions in a multicolor Pólya urn. J. Appl. Probab. 34 426–435.
  • Hall, P. and Heyde, C. C. (1980). Martingale Limit Theory and Its Application. Academic Press, New York.
  • Kushner, H. J. and Clark, D. S. (1978). Stochastic Approximation Methods for Constrained and Unconstrained Systems. Applied Mathematical Sciences 26. Springer, New York.
  • Laruelle, S. and Pagès, G. (2013). Randomized urn models revisited using stochastic approximation. Ann. Appl. Probab. 23 1409–1436.
  • Meyer, C. (2000). Matrix Analysis and Applied Linear Algebra. SIAM, Philadelphia, PA.
  • Meyer, C. D. (2015). Continuity of the Perron root. Linear Multilinear Algebra 63 1332–1336.
  • Pólya, G. (1930). Sur quelques points de la théorie des probabilités. Ann. Inst. Henri Poincaré 1 117–161.
  • Pólya, G. and Eggenberger, F. (1923). Über die Statistik verketteter Vorgänge. Z. Agnew. Math. Mech. 3 279–289.
  • Renlund, H. (2010). Generalized Polya urns via stochastic approximation. Preprint. Available at arxiv:1002.3716.
  • Renlund, H. (2011). Limit theorems for stochastic approximation algorithms. Preprint. Available at arxiv:1102.4741.
  • Resnick, S. (1992). Adventures in Stochastic Processes. Birkhäuser, Boston, MA.
  • Robbins, H. and Monro, S. (1951). A stochastic approximation method. Ann. Math. Stat. 22 400–407.
  • Seneta, E. (2006). Non-negative Matrices and Markov Chains. Springer Series in Statistics. Springer, New York.
  • Stroock, D. W. and Varadhan, S. R. S. (2006). Multidimensional Diffusion Processes. Classics in Mathematics. Springer, Berlin. Reprint of the 1997 edition.
  • Zhang, L. (2012). The Gaussian approximation for generalized Friedman’s urn model with heterogeneous and unbalanced updating. Sci. China Math. 55 2379–2404.
  • Zhang, L.-X. (2016). Central limit theorems of a recursive stochastic algorithm with applications to adaptive designs. Ann. Appl. Probab. 26 3630–3658.