Open Access
June 2020 Concentration of tempered posteriors and of their variational approximations
Pierre Alquier, James Ridgway
Ann. Statist. 48(3): 1475-1497 (June 2020). DOI: 10.1214/19-AOS1855

Abstract

While Bayesian methods are extremely popular in statistics and machine learning, their application to massive data sets is often challenging, when possible at all. The classical MCMC algorithms are prohibitively slow when both the model dimension and the sample size are large. Variational Bayesian methods aim at approximating the posterior by a distribution in a tractable family $\mathcal{F}$. Thus, MCMC are replaced by an optimization algorithm which is orders of magnitude faster. VB methods have been applied in such computationally demanding applications as collaborative filtering, image and video processing or NLP to name a few. However, despite nice results in practice, the theoretical properties of these approximations are not known. We propose a general oracle inequality that relates the quality of the VB approximation to the prior $\pi $ and to the structure of $\mathcal{F}$. We provide a simple condition that allows to derive rates of convergence from this oracle inequality. We apply our theory to various examples. First, we show that for parametric models with log-Lipschitz likelihood, Gaussian VB leads to efficient algorithms and consistent estimators. We then study a high-dimensional example: matrix completion, and a nonparametric example: density estimation.

Citation

Download Citation

Pierre Alquier. James Ridgway. "Concentration of tempered posteriors and of their variational approximations." Ann. Statist. 48 (3) 1475 - 1497, June 2020. https://doi.org/10.1214/19-AOS1855

Information

Received: 1 June 2018; Revised: 1 April 2019; Published: June 2020
First available in Project Euclid: 17 July 2020

zbMATH: 07241599
MathSciNet: MR4124331
Digital Object Identifier: 10.1214/19-AOS1855

Subjects:
Primary: 62G15
Secondary: 62C10 , 62C20 , 62F25 , 62G25

Keywords: Concentration of the posterior , PAC-Bayesian bounds , rate of convergence , variational approximation

Rights: Copyright © 2020 Institute of Mathematical Statistics

Vol.48 • No. 3 • June 2020
Back to Top