February 2024 Adaptive variational Bayes: Optimality, computation and applications
Ilsang Ohn, Lizhen Lin
Author Affiliations +
Ann. Statist. 52(1): 335-363 (February 2024). DOI: 10.1214/23-AOS2349


In this paper, we explore adaptive inference based on variational Bayes. Although several studies have been conducted to analyze the contraction properties of variational posteriors, there is still a lack of a general and computationally tractable variational Bayes method that performs adaptive inference. To fill this gap, we propose a novel adaptive variational Bayes framework, which can operate on a collection of models. The proposed framework first computes a variational posterior over each individual model separately and then combines them with certain weights to produce a variational posterior over the entire model. It turns out that this combined variational posterior is the closest member to the posterior over the entire model in a predefined family of approximating distributions. We show that the adaptive variational Bayes attains optimal contraction rates adaptively under very general conditions. We also provide a methodology to maintain the tractability and adaptive optimality of the adaptive variational Bayes even in the presence of an enormous number of individual models, such as sparse models. We apply the general results to several examples, including deep learning and sparse factor models, and derive new and adaptive inference results. In addition, we characterize an implicit regularization effect of variational Bayes and show that the adaptive variational posterior can utilize this.

Funding Statement

We acknowledge the generous support of NSF Grants DMS CAREER 1654579 and DMS 2113642. The first author was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korea government (MSIT) (NRF-2022R1F1A1069695) and INHA UNIVERSITY Research Grant.


We are very grateful to the Editor, the Associate Editor and three reviewers for their valuable comments, which have led to substantial improvement in our paper. We would like to thank Minwoo Chae, Kyoungjae Lee, Cheng Li and Ryan Martin for their helpful comments and suggestions.


Download Citation

Ilsang Ohn. Lizhen Lin. "Adaptive variational Bayes: Optimality, computation and applications." Ann. Statist. 52 (1) 335 - 363, February 2024. https://doi.org/10.1214/23-AOS2349


Received: 1 October 2022; Revised: 1 September 2023; Published: February 2024
First available in Project Euclid: 7 March 2024

MathSciNet: MR4718418
Digital Object Identifier: 10.1214/23-AOS2349

Primary: 62C10
Secondary: 62G20

Keywords: adaptive inference , Deep neural networks , model selection consistency , Posterior contraction rates , quasi-posteriors , variational Bayes

Rights: Copyright © 2024 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.52 • No. 1 • February 2024
Back to Top