Electronic Journal of Statistics

Improved Laplace approximation for marginal likelihoods

Erlis Ruli, Nicola Sartori, and Laura Ventura

Full-text: Open access

Abstract

Statistical applications often involve the calculation of intractable multidimensional integrals. The Laplace formula is widely used to approximate such integrals. However, in high-dimensional or small sample size problems, the shape of the integrand function may be far from that of the Gaussian density, and thus the standard Laplace approximation can be inaccurate. We propose an improved Laplace approximation that reduces the asymptotic error of the standard Laplace formula by one order of magnitude, thus leading to third-order accuracy. We also show, by means of practical examples of various complexity, that the proposed method is extremely accurate, even in high dimensions, improving over the standard Laplace formula. Such examples also demonstrate that the accuracy of the proposed method is comparable with that of other existing methods, which are computationally more demanding. An R implementation of the improved Laplace approximation is also provided through the R package iLaplace available on CRAN.

Article information

Source
Electron. J. Statist., Volume 10, Number 2 (2016), 3986-4009.

Dates
Received: January 2016
First available in Project Euclid: 29 December 2016

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1483002043

Digital Object Identifier
doi:10.1214/16-EJS1218

Mathematical Reviews number (MathSciNet)
MR3590640

Zentralblatt MATH identifier
1357.62129

Keywords
Asymptotic expansions for integrals Bayes factor conditional minimisation integrated likelihood normalising constant numerical integration

Citation

Ruli, Erlis; Sartori, Nicola; Ventura, Laura. Improved Laplace approximation for marginal likelihoods. Electron. J. Statist. 10 (2016), no. 2, 3986--4009. doi:10.1214/16-EJS1218. https://projecteuclid.org/euclid.ejs/1483002043


Export citation

References

  • [1] Azzalini, A. and Capitanio, A. (2003). Distributions generated by perturbation of symmetry with emphasis on a multivariate skew t-distribution., Journal of the Royal Statistical Society: Series B 65 367–389.
  • [2] Barndorff-Nielsen, O. E. and Cox, D. R. (1994)., Inference and Asymptotics. Chapman & Hall/CRC, London.
  • [3] Bates, D. M. and Watts, D. G. (1988)., Nonlinear Regression Analysis and Its Applications. Wiley Online Library, New York.
  • [4] Bellio, R. and Varin, C. (2005). A pairwise likelihood approach to generalized linear models with crossed random effects., Statistical Modelling 5 217–227.
  • [5] Bleistein, N. and Handelsman, R. (1986)., Asymptotic Expansions of Integrals. Dover, New York.
  • [6] Booth, J. G. and Hobert, J. P. (1999). Maximizing generalized linear mixed model likelihoods with an automated Monte Carlo EM algorithm., Journal of the Royal Statistical Society: Series B 61 265–285.
  • [7] Breslow, N. E. and Clayton, D. G. (1993). Approximate inference in generalized linear mixed models., Journal of the American Statistical Association 88 9–25.
  • [8] Butler, R. W. and Wood, A. T. A. (2002). Laplace approximations for hypergeometric functions with matrix argument., The Annals of Statistics 30 1155–1177.
  • [9] Chib, S. (1995). Marginal likelihood from the Gibbs output., Journal of the American Statistical Association 90 1313–1321.
  • [10] Chib, S. and Jeliazkov, I. (2001). Marginal likelihood from the Metropolis–Hastings output., Journal of the American Statistical Association 96 270–281.
  • [11] Cox, D. R. and Wermuth, N. (1990). An approximation to maximum likelihood estimates in reduced models., Biometrika 77 747–761.
  • [12] Davison, A. C., Fraser, D. A. S. and Reid, N. (2006). Improved likelihood inference for discrete data., Journal of the Royal Statistical Society: Series B 68 495–508.
  • [13] Diciccio, T. J. and Young, G. A. (2008). Conditional properties of unconditional parametric bootstrap procedures for inference in exponential families., Biometrika 95 747–758.
  • [14] DiCiccio, T. J., Kass, R. E., Raftery, A. and Wasserman, L. (1997). Computing Bayes factors by combining simulation and asymptotic approximations., Journal of the American Statistical Association 92 903–915.
  • [15] Diggle, P. J., Tawn, J. A. and Moyeed, R. A. (1998). Model-based geostatistics., Journal of the Royal Statistical Society: Series C 47 299–350.
  • [16] Evans, M. and Swartz, T. (2000)., Approximating Integrals via Monte Carlo and Deterministic Methods. Oxford University Press, Oxford.
  • [17] Ferkingstad, E. and Rue, H. (2015). Improving the INLA approach for approximate Bayesian inference for latent Gaussian models., Electronic Journal of Statistics 9 2706–2731.
  • [18] Fonseca, T. C., Ferreira, M. A. R. and Migon, H. S. (2008). Objective Bayesian analysis for the Student-$t$ regression model., Biometrika 95 325–333.
  • [19] Fournier, D. A., Skaug, H. J., Ancheta, J., Ianelli, J., Magnusson, A., Maunder, M. N., Nielsen, A. and Sibert, J. (2012). AD Model Builder: using automatic differentiation for statistical inference of highly parameterized complex nonlinear models., Optimization Methods and Software 27 233–249.
  • [20] Gelman, A. (2006). Prior distributions for variance parameters in hierarchical models., Bayesian Analysis 1 515–534.
  • [21] Hsiao, C. K., Huang, S.-Y. and Chang, C.-W. (2004). Bayesian marginal inference via candidate’s formula., Statistics and Computing 14 59–66.
  • [22] Jones, M. C. (2002). Marginal replacement in multivariate densities, with application to skewing spherically symmetric distributions., Journal of Multivariate Analysis 81 85–99.
  • [23] Karim, M. R. and Zeger, S. L. (1992). Generalized linear models with random effects; salamander mating revisited., Biometrics 48 631.
  • [24] Kass, R. E. and Raftery, A. E. (1995). Bayes factors., Journal of the American Statistical Association 90 773–795.
  • [25] Kass, R. E., Tierney, L. and Kadane, J. B. (1990). The validity of posterior expansions based on Laplace’s method. In, Bayesian and Likelihood Methods in Statistics and Econometrics: Essays in Honor of George A. Barnard (S. Geisser, J. Hodges, S. Press and A. Zellner, eds.) 473–487. North Holland.
  • [26] Kharroubi, S. A. and Sweeting, T. J. (2016). Exponential tilting in Bayesian asymptotics., Biometrika 103 337–349.
  • [27] Lindley, D. V. (1980). Approximate Bayesian methods., Trabajos de Estadistica Y de Investigacion Operativa 31 223–245.
  • [28] Martino, S., Akerkar, R. and Rue, H. (2011). Approximate Bayesian inference for survival models., Scandinavian Journal of Statistics 38 514–528.
  • [29] McCullagh, P. and Nelder, J. A. (1989)., Generalized Linear Models, 2 ed. Chapman and Hall, London.
  • [30] Nott, D. J., Fielding, M. and Leonte, D. (2009). On a generalization of the Laplace approximation., Statistics & Probability Letters 79 1397–1403.
  • [31] Ormerod, J. T. and Wand, M. P. (2010). Explaining variational approximations., The American Statistician 64 140–153.
  • [32] Pace, L., Salvan, A. and Ventura, L. (2006). Likelihood-based discrimination between separate scale and regression models., Journal of Statistical Planning and Inference 136 3539–3553.
  • [33] Plummer, M. (2013). JAGS version 3.4.0., http://mcmc-jags.sourceforge.net.
  • [34] Raudenbush, S. W., Yang, M.-L. and Yosef, M. (2000). Maximum likelihood for generalized linear models with nested random effects via high-order, multivariate Laplace approximation., Journal of Computational and Graphical Statistics 9 141–157.
  • [35] Rizopoulos, D., Verbeke, G. and Lesaffre, E. (2009). Fully exponential Laplace approximations for the joint modelling of survival and longitudinal data., Journal of the Royal Statistical Society: Series B 71 637–654.
  • [36] Rue, H., Martino, S. and Chopin, N. (2009). Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations., Journal of the Royal Statistical Society: Series B 71 319–392.
  • [37] Rue, H., Steinsland, I. and Erland, S. (2004). Approximating hidden Gaussian Markov random fields., Journal of the Royal Statistical Society: Series B 66 877–892.
  • [38] Ruli, E., Sartori, N. and Ventura, L. (2014). Marginal posterior simulation via higher-order tail area approximations., Bayesian Analysis 9 129–146.
  • [39] Ruli, E., Sartori, N. and Ventura, L. (2016). iLaplace: Improved Laplace Approximation for Integrals of Unimodal Functions R package version, 1.1.0.
  • [40] Shun, Z. (1997). Another look at the salamander mating data: A modified Laplace approximation approach., Journal of the American Statistical Association 92 341–349.
  • [41] Shun, Z. and McCullagh, P. (1995). Laplace approximation of high dimensional integrals., Journal of the Royal Statistical Society: Series B 57 749–760.
  • [42] Small, C. G. (2010)., Expansions and Asymptotics for Statistics. Chapman & Hall/CRC, Boca Ranton, Florida.
  • [43] Sung, Y. J. and Geyer, C. J. (2007). Monte Carlo likelihood inference for missing data models., The Annals of Statistics 35 990–1011.
  • [44] R Core Team (2016). R: A Language and Environment for Statistical Computing R Foundation for Statistical Computing, Vienna, Austria.
  • [45] Tierney, L. and Kadane, J. B. (1986). Accurate approximations for posterior moments and marginal densities., Journal of the American Statistical Association 81 82–86.