Adaptive importance sampling on discrete Markov chains

Abstract

In modeling particle transport through a medium, the path of a particle behaves as a transient Markov chain. We are interested in characteristics of the particle's movement conditional on its starting state, which take the form of a "score" accumulated with each transition. Importance sampling is an essential variance reduction technique in this setting, and we provide an adaptive (iteratively updated) importance sampling algorithm that converges exponentially to the solution. Examples illustrating this phenomenon are provided.