Bayesian Analysis

Laplace Approximation for Logistic Gaussian Process Density Estimation and Regression

Jaakko Riihimäki and Aki Vehtari

Full-text: Open access


Logistic Gaussian process (LGP) priors provide a flexible alternative for modelling unknown densities. The smoothness properties of the density estimates can be controlled through the prior covariance structure of the LGP, but the challenge is the analytically intractable inference. In this paper, we present approximate Bayesian inference for LGP density estimation in a grid using Laplace’s method to integrate over the non-Gaussian posterior distribution of latent function values and to determine the covariance function parameters with type-II maximum a posteriori (MAP) estimation. We demonstrate that Laplace’s method with MAP is sufficiently fast for practical interactive visualisation of 1D and 2D densities. Our experiments with simulated and real 1D data sets show that the estimation accuracy is close to a Markov chain Monte Carlo approximation and state-of-the-art hierarchical infinite Gaussian mixture models. We also construct a reduced-rank approximation to speed up the computations for dense 2D grids, and demonstrate density regression with the proposed Laplace approach.

Article information

Bayesian Anal., Volume 9, Number 2 (2014), 425-448.

First available in Project Euclid: 26 May 2014

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Gaussian process logistic transformation density estimation density regression approximate inference Laplace’s method


Riihimäki, Jaakko; Vehtari, Aki. Laplace Approximation for Logistic Gaussian Process Density Estimation and Regression. Bayesian Anal. 9 (2014), no. 2, 425--448. doi:10.1214/14-BA872.

Export citation


  • Adams, R. P. (2009). “Kernel Methods for Nonparametric Bayesian Inference of Probabilities and Point Processes.” Ph.D. thesis, University of Cambridge.
  • Adams, R. P., Murray, I., and MacKay, D. J. (2009). “The Gaussian process density sampler.” In Advances in Neural Information Processing Systems 21, 9–16. NIPS foundation.
  • Brooks, S. P. and Gelman, A. (1998). “General Methods for Monitoring Convergence of Iterative Simulations.” Journal of Computational and Graphical Statistics, 7(4): 434–455.
  • Cseke, B. and Heskes, T. (2011). “Approximate marginals in latent Gaussian models.” Journal of Machine Learning Research, 12: 417–454.
  • Cunningham, J. P., Shenoy, K. V., and Sahani, M. (2008). “Fast Gaussian process methods for point process intensity estimation.” In McCallum, A. and Roweis, S. (eds.), Proceedings of the 25th Annual International Conference on Machine Learning (ICML 2008), 192–199. Omnipress.
  • Gelman, A. (2006). “Prior distributions for variance parameters in hierarchical models.” Bayesian Analysis, 1(3): 515–533.
  • Geweke, J. (1989). “Bayesian Inference in Econometric Models Using Monte Carlo Integration.” Econometrica, 57(6): 1317–1339.
  • Geyer, C. J. (1992). “Practical Markov chain Monte Carlo.” Statistical Science, 7(4): 473–483.
  • Griebel, M. and Hegland, M. (2010). “A Finite Element Method for Density Estimation with Gaussian Process Priors.” SIAM Journal of Numerical Analysis, 47(6): 4759–4792.
  • Griffin, J. (2010). “Default priors for density estimation with mixture models.” Bayesian Analysis, 5(1): 45–64.
  • Harville, D. A. (1997). Matrix Algebra From a Statistician’s Perspective. Springer-Verlag.
  • Hoffmann, M. and Gelman, A. (2013). “The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo.” Journal of Machine Learning Research. In press.
  • Jylänki, P., Vanhatalo, J., and Vehtari, A. (2011). “Robust Gaussian Process Regression with a Student-t Likelihood.” Journal of Machine Learning Research, 12: 3227–3257.
  • Kong, A., Liu, J. S., and Wong, W. H. (1994). “Sequential Imputations and Bayesian Missing Data Problems.” Journal of the American Statistical Association, 89(425): 278–288.
  • Kundu, S. and Dunson, D. B. (2011). “Latent Factor Models for Density Estimation.” ArXiv:1108.2720v2.
  • Lenk, P. J. (1991). “Towards a Practicable Bayesian Nonparametric Density Estimator.” Biometrika, 78(3): 531–543.
  • — (2003). “Bayesian Semiparametric Density Estimation and Model Verification Using a Logistic-Gaussian Process.” Journal of Computational and Graphical Statistics, 12(3): 548–565.
  • Leonard, T. (1978). “Density Estimation, Stochastic Processes, and Prior Information.” Journal of the Royal Statistical Society: Series B (Methodological), 40(2): 113–146.
  • Neal, R. (1998). “Regression and Classification Using Gaussian Process Priors.” In Bernardo, J. M., Berger, J. O., David, A. P., and Smith, A. P. M. (eds.), Bayesian Statistics 6, 475–501. Oxford University Press.
  • Nickisch, H. and Rasmussen, C. E. (2008). “Approximations for Binary Gaussian Process Classification.” Journal of Machine Learning Research, 9: 2035–2078.
  • O’Hagan, A. (1978). “Curve fitting and optimal design for prediction.” Journal of the Royal Statistical Society. Series B (Methodological), 40(1): 1–42.
  • Quiñonero-Candela, J. and Rasmussen, C. E. (2005). “A Unifying View of Sparse Approximate Gaussian Process Regression.” Journal of Machine Learning Research, 6(3): 1939–1959.
  • Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. Cambridge: MIT Press.
  • Riihimäki, J., Jylänki, P., and Vehtari, A. (2013). “Nested Expectation Propagation for Gaussian Process Classification with a Multinomial Probit Likelihood.” Journal of Machine Learning Research, 14: 75–109.
  • Rubin, D. B. (1981). “The Bayesian Bootstrap.” Annals of Statistics, 9(1): 130–134.
  • Rue, H., Martino, S., and Chopin, N. (2009). “Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations.” Journal of the Royal Statistical Society. Series B (Statistical Methodology), 71(2): 319–392.
  • Snelson, E. and Ghahramani, Z. (2006). “Sparse Gaussian Processes Using Pseudo-Inputs.” In Weiss, Y., Schölkopf, B., and Platt, J. (eds.), Advances in Neural Information Processing Systems 18. The MIT Press.
  • Thorburn, D. (1986). “A Bayesian Approach to Density Estimation.” Biometrika, 73(1): 65–75.
  • Tokdar, S. T. (2007). “Towards a Faster Implementation of Density Estimation With Logistic Gaussian Process Priors.” Journal of Computational and Graphical Statistics, 16(3): 633–655.
  • Tokdar, S. T. and Ghosh, J. K. (2007). “Posterior consistency of logistic Gaussian process priors in density estimation.” Journal of Statistical Planning and Inference, 137(1): 34–42.
  • Tokdar, S. T., Zhu, Y. M., and Ghosh, J. K. (2010). “Bayesian Density Regression with Logistic Gaussian Process and Subspace Projection.” Bayesian Analysis, 5(2): 319–344.
  • van der Vaart, A. W. and van Zanten, J. H. (2009). “Adaptive Bayesian estimation using a Gaussian random field with inverse Gamma bandwidth.” The Annals of Statistics, 37(5B): 2655–2675.
  • Vanhatalo, J., Pietiläinen, V., and Vehtari, A. (2010). “Approximate inference for disease mapping with sparse Gaussian processes.” Statistics in Medicine, 29(15): 1580–1607.
  • Vanhatalo, J., Riihimäki, J., Hartikainen, J., Jylänki, P., Tolvanen, V., and Vehtari, A. (2013). “GPstuff: Bayesian Modeling with Gaussian Processes.” Journal of Machine Learning Research, 14: 1175–1179.
  • Vehtari, A. and Lampinen, J. (2002). “Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities.” Neural Computation, 14(10): 2439–2468.
  • Villani, M. and Larsson, R. (2006). “The Multivariate Split Normal Distribution and Asymmetric Principal Components Analysis.” Communications in Statistics: Theory & Methods, 35(6): 1123–1140.
  • Williams, C. K. I. and Barber, D. (1998). “Bayesian classification with Gaussian processes.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12): 1342–1351.