The Annals of Statistics

On Monte Carlo methods for estimating ratios of normalizing constants

Ming-Hui Chen and Qi-Man Shao

Full-text: Open access

Abstract

Recently, estimating ratios of normalizing constants has played an important role in Bayesian computations. Applications of estimating ratios of normalizing constants arise in many aspects of Bayesian statistical inference. In this article, we present an overview and discuss the current Monte Carlo methods for estimating ratios of normalizing constants. Then we propose a new ratio importance sampling method and establish its theoretical framework. We find that the ratio importance sampling method can be better than the current methods, for example, the bridge sampling method (Meng and Wong) and the path sampling method (Gelman and Meng), in the sense of minimizing asymptotic relative mean-square errors of estimators. An example is given for illustrative purposes. Finally, we present two special applications and the general implementation issues for estimating ratios of normalizing constants.

Article information

Source
Ann. Statist., Volume 25, Number 4 (1997), 1563-1594.

Dates
First available in Project Euclid: 9 September 2002

Permanent link to this document
https://projecteuclid.org/euclid.aos/1031594732

Digital Object Identifier
doi:10.1214/aos/1031594732

Mathematical Reviews number (MathSciNet)
MR1463565

Zentralblatt MATH identifier
0936.62028

Subjects
Primary: 62E25
Secondary: 62A15 62A99: None of the above, but in this section

Keywords
Bayesian computation bridge sampling Gibbs sampling importance sampling Markov chain Monte Carlo Metropolis-Hastings algorithm path sampling ratio importance sampling

Citation

Chen, Ming-Hui; Shao, Qi-Man. On Monte Carlo methods for estimating ratios of normalizing constants. Ann. Statist. 25 (1997), no. 4, 1563--1594. doi:10.1214/aos/1031594732. https://projecteuclid.org/euclid.aos/1031594732


Export citation

References

  • BERGER, J. O. and PERICCHI, L. R. 1996. The intrinsic Bay es factor for model selection and prediction. J. Amer. Statist. Assoc. 91 109 122. Z.
  • CHEN, M.-H. 1994a. Simulating ratios of normalization constants for the Gibbs Sampler. Technical report, Dept. Mathematical Sciences, Worcester Poly technic Inst.
  • CHEN, M.-H. 1994b. Importance-weighted marginal Bayesian posterior density estimation. J. Amer. Statist. Assoc. 89 818 824. Z.
  • CHEN, M.-H. and SCHMEISER, B. W. 1993. Performance of the Gibbs, hit-and-run, and Metropolis samplers. J. Comput. Graph. Statist. 2 251 272. Z.
  • CHIB, S. 1995. Marginal likelihood from the Gibbs output. J. Amer. Statist. Assoc. 90 1313 1321. Z.
  • DEVROy E, L. 1986. Non-Uniform Random Variate Generation. Springer, New York. Z.
  • GELFAND, A. E. and SMITH, A. F. M. 1990. Sampling based approaches to calculating marginal densities. J. Amer. Statist. Assoc. 85 398 409. Z.
  • GELFAND, A. E., SMITH, A. F. M. and LEE, T. M. 1992. Bayesian analysis of constrained parameter and truncated data problems using Gibbs sampling. J. Amer. Statist. Assoc. 87 523 532. Z.
  • GELMAN, A. and MENG, X.-L. 1994. Path sampling for computing normalizing constants: identities and theory. Technical Report 377, Dept. Statistics, Univ. Chicago. Z.
  • GELMAN, A. and MENG, X.-L. 1996. Simulating normalizing constants: from importance sampling to bridge sampling to path sampling. Technical Report 440, Dept. Statistics, Univ. Chicago. Z.
  • GEMAN, S. and GEMAN, D. 1984. Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analy sis and Machine Intelligence 6 721 741. Z.
  • GEWEKE, J. 1989. Bayesian inference in econometrics models using Monte Carlo integration. Econometrica 57 1317 1340. Z.
  • GEWEKE, J. 1994. Bayesian comparison of econometric models. Technical Report 532, Federal Reserve Bank of Minneapolis and Univ. Minnesota. Z.
  • GEy ER, C. J. 1994. Estimating normalizing constants and reweighting mixtures in Markov chain Monte Carlo. Rev. Technical Report 568, School of Statistics, Univ. Minnesota. Z.
  • GREEN, P. J. 1992. Discussion of ``Constrained Monte Carlo maximum likelihood for dependent data,'' by C. J. Gey er and E. A. Thompson. J. Roy. Statist. Soc. Ser. B 54 657 699. Z.
  • HASTINGS, W. K. 1970. Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57 97 109. Z.
  • HE, X. and SHAO, Q. M. 1996. A general Bahadur representation of M-estimators and its application to linear regression with nonstochastic designs. Ann. Statist. 24 2608 2630. Z.
  • JANSSEN, P., JURECKOVA, J. and VERAVERBEKE, N. 1985. Rate of convergence of oneand two-step M-estimators with applications to maximum likelihood and Pitman estimators. Ann. Statist. 13 1222 1229. Z.
  • MENG, X. L. and WONG, W. H. 1996. Simulating ratios of normalizing constants via a simple identity: a theoretical exploration. Statist. Sinica 6 831 860. Z.
  • METROPOLIS, N., ROSENBLUTH, A. W., ROSENBLUTH, M. N., TELLER, A. H. and TELLER, E. 1953. Equations of state calculations by fast computing machines. J. Chem. Phy s. 21 1087 1092. Z.
  • MULLER, P. 1991. A generic approach to posterior integration and Gibbs sampling. Technical ¨ Report 91-09, Dept. Statistics, Purdue Univ. Z.
  • TANNER, M. A. and WONG, W. H. 1987. The calculation of posterior distributions by data augmentation. J. Amer. Statist. Assoc. 82 528 549. Z. Z.
  • TIERNEY, L. 1994. Markov chains for exploring posterior distributions with discussion. Ann. Statist. 22 1701 1762.
  • WORCESTER, MASSACHUSETTS 01609-2280 EUGENE, OREGON 97403-1222 E-MAIL: mhchen@wpi.edu E-MAIL: qmshao@darkwing.uoregon.edu