Electronic Journal of Statistics

A variational Bayes approach to a semiparametric regression using Gaussian process priors

Victor M. H. Ong, David K. Mensah, David J. Nott, Seongil Jo, Beomjo Park, and Taeryon Choi

Full-text: Open access


This paper presents a variational Bayes approach to a semiparametric regression model that consists of parametric and nonparametric components. The assumed univariate nonparametric component is represented with a cosine series based on a spectral analysis of Gaussian process priors. Here, we develop fast variational methods for fitting the semiparametric regression model that reduce the computation time by an order of magnitude over Markov chain Monte Carlo methods. Further, we explore the possible use of the variational lower bound and variational information criteria for model choice of a parametric regression model against a semiparametric alternative. In addition, variational methods are developed for estimating univariate shape-restricted regression functions that are monotonic, monotonic convex or monotonic concave. Since these variational methods are approximate, we explore some of the trade-offs involved in using them in terms of speed, accuracy and automation of the implementation in comparison with Markov chain Monte Carlo methods and discuss their potential and limitations.

Article information

Electron. J. Statist., Volume 11, Number 2 (2017), 4258-4296.

Received: August 2016
First available in Project Euclid: 8 November 2017

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G08: Nonparametric regression
Secondary: 62F15: Bayesian inference

Cosine series Gaussian process model selection shape restricted regression variational Bayes

Creative Commons Attribution 4.0 International License.


Ong, Victor M. H.; Mensah, David K.; Nott, David J.; Jo, Seongil; Park, Beomjo; Choi, Taeryon. A variational Bayes approach to a semiparametric regression using Gaussian process priors. Electron. J. Statist. 11 (2017), no. 2, 4258--4296. doi:10.1214/17-EJS1324. https://projecteuclid.org/euclid.ejs/1510111112

Export citation


  • Adler, R. J. and Taylor, J. E. (2007)., Random fields and geometry. Springer Monographs in Mathematics. Springer, New York.
  • Attias, H. (2000). A variational Bayesian framework for graphical models. In, Advances in Neural Information Processing Systems 12 209–215. MIT Press.
  • Cai, B. and Dunson, D. B. (2007). Bayesian multivariate isotonic regression splines: applications to carcinogenicity studies., J. Amer. Statist. Assoc. 102 1158–1171.
  • Chakraborty, S., Ghosh, M. and Mallick, B. K. (2012). Bayesian nonlinear regression for large $p$ small $n$ problems., J. Multivariate Anal. 108 28–40.
  • Cressie, N. and Wikle, C. K. (2011)., Statistics for spatio-temporal data. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., Hoboken, NJ.
  • Curtis, S. M. and Ghosh, S. K. (2011). A variable selection approach to monotonic regression with Bernstein polynomials., J. Appl. Stat. 38 961–976.
  • Gelfand, A. E. and Dey, D. K. (1994). Bayesian model choice: asymptotics and exact calculations., J. Roy. Statist. Soc. Ser. B 56 501–514.
  • Ghahramani, Z. and Beal, M. J. (2001). Propagation algorithms for variational Bayesian learning., Advances in neural information processing systems 13 507–513.
  • Giordano, R. J., Broderick, T. and Jordan, M. I. (2015). Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes. In, Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama andR. Garnett, eds.) 1441–1449.
  • Goldsmith, J., Wand, M. P. and Crainiceanu, C. (2011). Functional regression via variation Bayes., Electron. J. Stat. 5 572–602.
  • Grenander, U. (1981)., Abstract inference. John Wiley & Sons, Inc., New York Wiley Series in Probability and Mathematical Statistics.
  • Hofner, B., Kneib, T. and Hothorn, T. (2016). A unified framework of constrained regression., Stat. Comput. 26 1–14.
  • Hu, Y., Zhao, K. and Lian, H. (2015). Bayesian quantile regression for partially linear additive models., Stat. Comput. 25 651–668.
  • Ji, C., Shen, H. and West, M. (2010). Bounded approximations for marginal likelihoods Technical Report No. 10-05, Institute of Decision Sciences, Duke, University.
  • Jo, S., Choi, T., Park, B. and Lenk, P. J. (2017). bsamGP: Bayesian Spectral Analysis Models using Gaussian Process Priors R package version, 1.0.2.
  • Jordan, M. I. (2004). Graphical models., Statist. Sci. 19 140–155.
  • Jordan, M., Ghahramani, Z., Jaakkola, T. and Saul, L. (1999). An Introduction to Variational Methods for Graphical Models., Machine Learning 37 183-233.
  • Knowles, D. A. and Minka, T. (2011). Non-conjugate Variational Message Passing for Multinomial and Binary Regression. In, Advances in Neural Information Processing Systems 24 (J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira andK. Q. Weinberger, eds.) 1701–1709. Curran Associates, Inc.
  • Ko, K., Qu, L. and Vannucci, M. (2009). Wavelet-based Bayesian estimation of partially linear regression models with long memory errors., Statist. Sinica 19 1463–1478.
  • Lázaro-Gredilla, M., Quiñonero-Candela, J., Rasmussen, C. E. and Figueiras-Vidal, A. R. (2010). Sparse spectrum Gaussian process regression., J. Mach. Learn. Res. 11 1865–1881.
  • Lenk, P. J. (1999). Bayesian inference for semiparametric regression using a Fourier representation., J. R. Stat. Soc. Ser. B Stat. Methodol. 61 863–879.
  • Lenk, P. J. and Choi, T. (2017). Bayesian analysis of shape-restricted functions using Gaussian process priors., Statist. Sinica 27 43–69.
  • Lin, L. and Dunson, D. B. (2014). Bayesian monotone regression using Gaussian process projection., Biometrika 101 303–317.
  • Luts, J., Broderick, T. and Wand, M. P. (2014). Real-time semiparametric regression., J. Comput. Graph. Statist. 23 589–615.
  • Luts, J. and Wand, M. P. (2015). Variational inference for count response semiparametric regression., Bayesian Anal. 10 991–1023.
  • Meyer, M. C., Hackstadt, A. J. and Hoeting, J. A. (2011). Bayesian estimation and inference for generalised partial linear models using shape-restricted splines., J. Nonparametr. Stat. 23 867–884.
  • Neville, S. E., Ormerod, J. T. and Wand, M. P. (2014). Mean field variational Bayes for continuous sparse signal shrinkage: pitfalls and remedies., Electron. J. Stat. 8 1113–1151.
  • Nott, D. J., Tan, S. L., Villani, M. and Kohn, R. (2012). Regression density estimation with variational methods and stochastic approximation., J. Comput. Graph. Statist. 21 797–820.
  • O’Hagan, A. (1978). Curve fitting and optimal design for prediction., J. Roy. Statist. Soc. Ser. B 40 1–42.
  • Ormerod, J. T. and Wand, M. P. (2010). Explaining variational approximations., Amer. Statist. 64 140–153.
  • Paciorek, C. J. (2007). Bayesian smoothing with Gaussian processes using Fourier basis functions in the spectralGP package., Journal of Statistical Software 19 2.
  • Paisley, J. W., Blei, D. M. and Jordan, M. I. (2012). Variational Bayesian Inference with Stochastic Search. In, Proceedings of the 29th International Conference on Machine Learning (ICML-12).
  • Rasmussen, C. E. and Williams, C. K. I. (2006)., Gaussian processes for machine learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA.
  • Riihimäki, J. and Vehtari, A. (2010). Gaussian processes with monotonicity information. In, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (Y. W. Teh andM. Titterington, eds.). Proceedings of Machine Learning Research 9 645–652.
  • Shively, T. S., Sager, T. W. and Walker, S. G. (2009). A Bayesian approach to non-parametric monotone function estimation., J. R. Stat. Soc. Ser. B Stat. Methodol. 71 159–175.
  • Tan, L. L., Ong, V. H., Nott, D. and Jasra, A. (2016). Variational inference for sparse spectrum Gaussian process regression., Stat. Comput. to appear.
  • Titterington, D. M. (2004). Bayesian methods for neural networks and related models., Statist. Sci. 19 128–139.
  • Turner, R. E. and Sahani, M. (2011). Two problems with variational expectation maximisation for time-series models. In, Bayesian Time series models (D. Barber, T. Cemgil andS. Chiappa, eds.) 5, 109–130. Cambridge University Press.
  • Waldmann, E. and Kneib, T. (2015). Variational approximations in geoadditive latent Gaussian regression: mean and quantile regression., Stat. Comput. 25 1247–1263.
  • Wand, M. P. (2014). Fully simplified multivariate normal updates in non-conjugate variational message passing., J. Mach. Learn. Res. 15 1351–1369.
  • Wand, M. P. and Ormerod, J. T. (2011). Penalized wavelets: embedding wavelets into semiparametric regression., Electron. J. Stat. 5 1654–1717.
  • Wang, X. and Berger, J. O. (2016). Estimating shape constrained functions using Gaussian processes., SIAM/ASA J. Uncertain. Quantif. 4 1–25.
  • Wang, B. and Titterington, D. M. (2004). Inadequacy of interval estimates corresponding to variational Bayesian approximations. In, In Workshop on Artificial Intelligence and Statistics 373–380.
  • Waterhouse, S., Mackay, D. and Robinson, T. (1996). Bayesian methods for mixture of experts. In, Advances in Neural Information Processing Systems 8 351–357. MIT Press.
  • Winn, J. and Bishop, C. M. (2005). Variational message passing., J. Mach. Learn. Res. 6 661–694.
  • Yatchew, A. (2003)., Semiparametric Regression for the Applied Econometrician. Cambridge University Press.
  • You, C., Ormerod, J. T. and Müller, S. (2014). On variational Bayes estimation and variational information criteria for linear regression models., Aust. N. Z. J. Stat. 56 73–87.
  • Zhang, Z., Wang, D., Dai, G. and Jordan, M. I. (2014). Matrix-variate Dirichlet process priors with applications., Bayesian Anal. 9 259–285.
  • Zhao, K. and Lian, H. (2014). Variational inferences for partially linear additive models with variable selection., Comput. Statist. Data Anal. 80 223–239.