## The Annals of Applied Probability

### Stochastic approximation with random step sizes and urn models with random replacement matrices having finite mean

#### Abstract

The stochastic approximation algorithm is a useful technique which has been exploited successfully in probability theory and statistics for a long time. The step sizes used in stochastic approximation are generally taken to be deterministic and same is true for the drift. However, the specific application of urn models with random replacement matrices motivates us to consider stochastic approximation in a setup where both the step sizes and the drift are random, but the sequence is uniformly bounded. The problem becomes interesting when the negligibility conditions on the errors hold only in probability. We first prove a result on stochastic approximation in this setup, which is new in the literature. Then, as an application, we study urn models with random replacement matrices.

In the urn model, the replacement matrices need neither be independent, nor identically distributed. We assume that the replacement matrices are only independent of the color drawn in the same round conditioned on the entire past. We relax the usual second moment assumption on the replacement matrices in the literature and require only first moment to be finite. We require the conditional expectation of the replacement matrix given the past to be close to an irreducible matrix, in an appropriate sense. We do not require any of the matrices to be balanced or nonrandom. We prove convergence of the proportion vector, the composition vector and the count vector in $L^{1}$, and hence in probability. It is to be noted that the related differential equation is of Lotka–Volterra type and can be analyzed directly.

#### Article information

Source
Ann. Appl. Probab., Volume 29, Number 4 (2019), 2033-2066.

Dates
First available in Project Euclid: 23 July 2019

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1563869036

Digital Object Identifier
doi:10.1214/18-AAP1441

Mathematical Reviews number (MathSciNet)
MR3984252

Zentralblatt MATH identifier
07120702

Subjects
Primary: 62L20: Stochastic approximation
Secondary: 60F15: Strong theorems 60G42: Martingales with discrete parameter

#### Citation

Gangopadhyay, Ujan; Maulik, Krishanu. Stochastic approximation with random step sizes and urn models with random replacement matrices having finite mean. Ann. Appl. Probab. 29 (2019), no. 4, 2033--2066. doi:10.1214/18-AAP1441. https://projecteuclid.org/euclid.aoap/1563869036

#### References

• Athreya, K. B. and Karlin, S. (1968). Embedding of urn schemes into continuous time Markov branching processes and related limit theorems. Ann. Math. Stat. 39 1801–1817.
• Athreya, K. B. and Ney, P. E. (2004). Branching Processes. Dover, Mineola, NY.
• Bai, Z.-D. and Hu, F. (2005). Asymptotics in randomized URN models. Ann. Appl. Probab. 15 914–940.
• Benaïm, M. (1999). Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII. Lecture Notes in Math. 1709 1–68. Springer, Berlin.
• Borkar, V. S. (2008). Stochastic Approximation. Cambridge Univ. Press, Cambridge.
• Dasgupta, A. and Maulik, K. (2011). Strong laws for urn models with balanced replacement matrices. Electron. J. Probab. 16 1723–1749.
• Freedman, D. A. (1965). Bernard Friedman’s urn. Ann. Math. Stat. 36 956–970.
• Gouet, R. (1997). Strong convergence of proportions in a multicolor Pólya urn. J. Appl. Probab. 34 426–435.
• Hall, P. and Heyde, C. C. (1980). Martingale Limit Theory and Its Application. Academic Press, New York.
• Kushner, H. J. and Clark, D. S. (1978). Stochastic Approximation Methods for Constrained and Unconstrained Systems. Applied Mathematical Sciences 26. Springer, New York.
• Laruelle, S. and Pagès, G. (2013). Randomized urn models revisited using stochastic approximation. Ann. Appl. Probab. 23 1409–1436.
• Meyer, C. (2000). Matrix Analysis and Applied Linear Algebra. SIAM, Philadelphia, PA.
• Meyer, C. D. (2015). Continuity of the Perron root. Linear Multilinear Algebra 63 1332–1336.
• Pólya, G. (1930). Sur quelques points de la théorie des probabilités. Ann. Inst. Henri Poincaré 1 117–161.
• Pólya, G. and Eggenberger, F. (1923). Über die Statistik verketteter Vorgänge. Z. Agnew. Math. Mech. 3 279–289.
• Renlund, H. (2010). Generalized Polya urns via stochastic approximation. Preprint. Available at arxiv:1002.3716.
• Renlund, H. (2011). Limit theorems for stochastic approximation algorithms. Preprint. Available at arxiv:1102.4741.
• Resnick, S. (1992). Adventures in Stochastic Processes. Birkhäuser, Boston, MA.
• Robbins, H. and Monro, S. (1951). A stochastic approximation method. Ann. Math. Stat. 22 400–407.
• Seneta, E. (2006). Non-negative Matrices and Markov Chains. Springer Series in Statistics. Springer, New York.
• Stroock, D. W. and Varadhan, S. R. S. (2006). Multidimensional Diffusion Processes. Classics in Mathematics. Springer, Berlin. Reprint of the 1997 edition.
• Zhang, L. (2012). The Gaussian approximation for generalized Friedman’s urn model with heterogeneous and unbalanced updating. Sci. China Math. 55 2379–2404.
• Zhang, L.-X. (2016). Central limit theorems of a recursive stochastic algorithm with applications to adaptive designs. Ann. Appl. Probab. 26 3630–3658.