Advances in Applied Probability

Analysis of a stochastic approximation algorithm for computing quasi-stationary distributions

J. Blanchet, P. Glynn, and S. Zheng

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

We study the convergence properties of a Monte Carlo estimator proposed in the physics literature to compute the quasi-stationary distribution on a transient set of a Markov chain (see De Oliveira and Dickman (2005), (2006), and Dickman and Vidigal (2002)). Using the theory of stochastic approximations we verify the consistency of the estimator and obtain an associated central limit theorem. We provide an example showing that convergence might occur very slowly if a certain eigenvalue condition is violated. We alleviate this problem using an easy-to-implement projection step combined with averaging.

Article information

Source
Adv. in Appl. Probab., Volume 48, Number 3 (2016), 792-811.

Dates
First available in Project Euclid: 19 September 2016

Permanent link to this document
https://projecteuclid.org/euclid.aap/1474296315

Mathematical Reviews number (MathSciNet)
MR3568892

Zentralblatt MATH identifier
1352.60106

Subjects
Primary: 60J22: Computational methods in Markov chains [See also 65C40]
Secondary: 60J10: Markov chains (discrete-time Markov processes on discrete state spaces)

Keywords
Quasi-stationary distribution stochastic approximation Markov chain central limit theorem

Citation

Blanchet, J.; Glynn, P.; Zheng, S. Analysis of a stochastic approximation algorithm for computing quasi-stationary distributions. Adv. in Appl. Probab. 48 (2016), no. 3, 792--811. https://projecteuclid.org/euclid.aap/1474296315


Export citation

References

  • Aldous, D., Flannery, B. and Palacios, J. L. (1988). Two applications of urn processes to the fringe analysis of search trees and the simulation of quasi-stationary distributions of Markov chains. Prob. Eng. Inf. Sci. 2, 293–307.
  • Athreya, K. B. and Karlin, S. (1968). Embedding of urn schemes into continuous time Markov branching processes and related limit theorems. Ann. Math. Statist. 39, 1801–1817.
  • Benaïm, M. and Cloez, B. (2015). A stochastic approximation approach to quasi-stationary distributions on finite spaces. Electron. Commun. Prob. 20, 14 pp.
  • Blanchet, J., Glynn, P. and Zheng, S. (2013). Empirical analysis of a stochastic approximation approach for computing quasi-stationary distributions. In EVOLVE, Springer, Berlin, 19–37.
  • Boyd, S. and Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press.
  • Burzdy, K., Hołyst, R. and March, P. (2000). A Fleming–Viot particle representation of the Dirichlet Laplacian. Commun. Math. Phys. 214, 679–703.
  • Darroch, J. N. and Seneta, E. (1965). On quasi-stationary distributions in absorbing discrete-time finite Markov chains. J. Appl. Prob. 2, 88–100.
  • Darroch, J. N. and Seneta, E. (1967). On quasi-stationary distributions in absorbing continuous-time finite Markov chains. J. Appl. Prob. 4, 192–196.
  • De Oliveira, M. M. and Dickman, R. (2005). How to simulate the quasistationary state. Phys. Rev. E 71, 016129.
  • De Oliveira, M. M. and Dickman, R. (2006). Quasi-stationary simulation: the subcritical contact process. Brazilian J. Phys. 36, 685–689.
  • Del Moral, P. and Miclo, L. (2006). Self-interacting Markov chains. Stoch. Anal. Appl. 24, 615–660.
  • Dickman, R. and Vidigal, R. (2002). Quasi-stationary distributions for stochastic processes with an absorbing state. J. Phys. A 35, 1147–1166.
  • Dimov, I. T., Karaivanova, A. N. and Yordanova, P. I. (1998). Monte Carlo algorithms for calculating eigenvalues. In Monte Carlo and Quasi-Monte Carlo Methods 1996 (Salzburg; Lecture Notes Statist. 127), Springer, New York, pp. 205–220.
  • Ferrari, P. A. and Marić, N. (2007). Quasistationary distributions and Fleming–Viot processes in countable spaces. Electron. J. Prob. 12, 684–702.
  • Golub, G. H. and Van Loan, C. F. (1996). Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore, MD.
  • Groisman, P. and Jonckheere, M. (2013). Simulation of quasi-stationary distributions on countable spaces. Markov. Process. Relat. Fields. 19, 521–542.
  • Karlin, S. and Taylor, H. M. (1975). A First Course in Stochastic Processes, 2nd edn. Academic Press, New York.
  • Krasulina, T. P. (2013). The method of stochastic approximation for the determination of the least eigenvalue of a symmetrical matrix. USSR Comput. Math. Math. Phys. 9, 189–195.
  • Krasulina, T. P. (1970). Method of stochastic approximation in the determination of the largest eigenvalue of the mathematical expectation of random matrices. Automat. Rem. Contr. 2, 215–221.
  • Kushner, H. J. and Yin, G. (2003). Stochastic Approximation and Recursive Algorithms and Applications, 2nd edn. Springer, New York.
  • Méléard, S. and Villemonais, D. (2012). Quasi-stationary distributions and population processes. Prob. Surveys 9, 340–410.
  • Oja, E. and Karhunen, J. (1985). On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. J. Math. Anal. Appl. 106, 69–84.
  • Permantle, R. (2007). A survey of random processes with reinforcement. Prob. Surveys 4, 1–79.
  • Polyak, B. T. and Juditsky, A. B. (1992). Acceleration of stochastic approximation by averaging. SIAM J. Control Optimization 30, 838–855.
  • Robbins, H. and Monro, S. (1951). A stochastic approximation method. Ann. Math. Statist. 22, 400–407.
  • Zheng, S. (2014). Stochastic approximation algorithms in the estimation of quasi-stationary distribution of finite and general state space Markov chains. Doctoral Thesis, Columbia University.