The Annals of Applied Probability

Dynamic importance sampling for uniformly recurrent Markov chains

Paul Dupuis and Hui Wang

Full-text: Open access


Importance sampling is a variance reduction technique for efficient estimation of rare-event probabilities by Monte Carlo. In standard importance sampling schemes, the system is simulated using an a priori fixed change of measure suggested by a large deviation lower bound analysis. Recent work, however, has suggested that such schemes do not work well in many situations. In this paper we consider dynamic importance sampling in the setting of uniformly recurrent Markov chains. By “dynamic” we mean that in the course of a single simulation, the change of measure can depend on the outcome of the simulation up till that time. Based on a control-theoretic approach to large deviations, the existence of asymptotically optimal dynamic schemes is demonstrated in great generality. The implementation of the dynamic schemes is carried out with the help of a limiting Bellman equation. Numerical examples are presented to contrast the dynamic and standard schemes.

Article information

Ann. Appl. Probab., Volume 15, Number 1A (2005), 1-38.

First available in Project Euclid: 28 January 2005

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 60F10: Large deviations 65C05: Monte Carlo methods 93E20: Optimal stochastic control

Asymptotic optimality importance sampling Markov chain Monte Carlo simulation rare events stochastic game weak convergence


Dupuis, Paul; Wang, Hui. Dynamic importance sampling for uniformly recurrent Markov chains. Ann. Appl. Probab. 15 (2005), no. 1A, 1--38. doi:10.1214/105051604000001016.

Export citation


  • Asmussen, S. (1985). Conjugate processes and the simulation of ruin probability. Stochastic Process. Appl. 20 213–229.
  • Asmussen, S. (1989). Risk theory in a Markovian environment. Scand. Actuar. J. 89 69–100.
  • Asmussen, S., Rubinstein, R. and Wang, C. L. (1994). Regenerative rare event simulation via likelihood ratios. J. Appl. Probab. 31 797–815.
  • Bertsekas, D. and Shreve, S. (1978). Stochastic Optimal Control: The Discrete Time Case. Academic Press, San Diego, CA.
  • Billingsley, P. (1995). Probability and Measure, 3rd ed. Wiley, New York.
  • Breiman, L. (1968). Probability Theory. Addison–Wesley, Reading, MA.
  • Bucklew, J. A. (1990). Large Deviations Techniques in Decision, Simulation and Estimation. Wiley, New York.
  • Bucklew, J. A. (1998). The blind simulation problem and regenerative processes. IEEE Trans. Inform. Theory 44 2877–2891.
  • Bucklew, J. A., Ney, P. and Sadowsky, J. S. (1990). Monte Carlo simulation and large deviations theory for uniformly recurrent Markov chains. J. Appl. Probab. 27 44–59.\goodbreak
  • Chen, J., Lu, D., Sadowsky, J. S. and Yao, K. (1993). On importance sampling in digital communications. Part I: Fundamentals. IEEE J. Selected Areas in Comm. 11 289–307.
  • Collamore, J. F. (2002). Importance sampling techniques for the multidimensional ruin problem for general Markov additive sequences of random vectors. Ann. Appl. Probab. 12 382–421.
  • Cottrel, M., Fort, J. C. and Malgouynes, G. (1983). Large deviations and rare events in the study of stochastic algorithms. IEEE Trans. Automat. Control AC-28 907–920.
  • de Acosta, A. (1990). Large deviations for empirical measures of Markov chains. J. Theoret. Probab. 3 395–431.
  • Dupuis, P. and Ellis, R. S. (1997). A Weak Convergence Approach to the Theory of Large Deviations. Wiley, New York.
  • Dupuis, P. and Kushner, H. J. (1987). Stochastic systems with small noise, analysis and simulation; a phase locked loop example. SIAM J. Appl. Math. 47 643–661.
  • Dupuis, P. and Wang, H. (2005). Importance sampling, large deviations, and differential games. Ann. Probab. To appear.
  • Glasserman, P. (2004). Monte Carlo Methods in Financial Engineering. Springer, New York.
  • Glasserman, P. and Kou, S. (1995). Analysis of an importance sampling estimator for tandem queues. ACM Trans. Modeling Comp. Simulation 4 22–42.
  • Glasserman, P. and Wang, Y. (1997). Counter examples in importance sampling for large deviations probabilities. Ann. Appl. Probab. 7 731–746.
  • Glynn, P. W. (1995). Large deviations for the infinite server queue in heavy traffic. IMA Vol. Math. Appl. 71 387–394.
  • Heidelberger, P. (1995). Fast simulation of rare events in queueing and reliability models. ACM Trans. Modeling Comp. Simulation 4 43–85.
  • Iscoe, I., Ney, P. and Nummelin, E. (1985). Large deviations of uniformly recurrent Markov additive processes. Adv. in Appl. Math. 6 373–412.
  • Kushner, H. J. (1990). Weak Convergence Methods and Singularly Perturbed Stochastic Control and Filtering Problems. Birkhäuser, Boston.
  • Lehtonen, T. and Nyhrinen, H. (1992). Simulating level crossing probabilities by importance sampling. Adv. in Appl. Probab. 24 858–874.
  • Lehtonen, T. and Nyhrinen, H. (1992). On asymptotically efficient simulating of ruin probabilities in a Markovian environment. Scand. Actuar. J. 1 60–75.
  • Ney, P. and Nummelin, E. (1987). Markov additive processes I: Eigenvalue properties and limit theorems. Ann. Probab. 15 561–592.
  • Ney, P. and Nummelin, E. (1987). Markov additive processes II: Large deviations. Ann. Probab. 15 593–609.
  • Rockafellar, R. T. (1970). Convex Analysis. Princeton Univ. Press.
  • Sadowsky, J. S. (1991). Large deviations and efficient simulation of excessive backlogs in a ${G}{I}/{G}/m$ queue. IEEE Trans. Automat. Control 36 1383–1394.
  • Sadowsky, J. S. (1993). On the optimality and stability of exponential twisting in Monte Carlo estimation. IEEE Trans. Inform. Theory 39 119–128.
  • Sadowsky, J. S. (1996). On Monte Carlo estimation of large deviations probabilities. Ann. Appl. Probab. 6 399–422.
  • Sadowsky, J. S. and Bucklew, J. A. (1990). On large deviations theory and asymptotically efficient Monte Carlo estimation. IEEE Trans. Inform. Theory 36 579–588.
  • Shahsbuddin, P. (1994). Importance sampling for the simulation of highly reliable Markovian systems. Management Sci. 40 333–352.
  • Siegmund, D. (1976). Importance sampling in the Monte Carlo study of sequential tests. Ann. Statist. 4 673–684.