A Bernstein-Von Mises Theorem for discrete probability distributions

Abstract

We investigate the asymptotic normality of the posterior distribution in the discrete setting, when model dimension increases with sample size. We consider a probability mass function θ₀ on ℕ∖{0} and a sequence of truncation levels (k_n)_n satisfying k_n³≤ninf _i≤knθ₀(i). Let θ̂ denote the maximum likelihood estimate of (θ₀(i))_i≤kn and let Δ_n(θ₀) denote the k_n-dimensional vector which i-th coordinate is defined by $\sqrt{n}(\hat{\theta}_{n}(i)-\theta_{0}(i))$ for 1≤i≤k_n. We check that under mild conditions on θ₀ and on the sequence of prior probabilities on the k_n-dimensional simplices, after centering and rescaling, the variation distance between the posterior distribution recentered around θ̂_n and rescaled by $\sqrt{n}$ and the k_n-dimensional Gaussian distribution $\mathcal{N}(\Delta_{n}(\theta_{0}),I^{-1}(\theta_{0}))$ converges in probability to 0. This theorem can be used to prove the asymptotic normality of Bayesian estimators of Shannon and Rényi entropies.

The proofs are based on concentration inequalities for centered and non-centered Chi-square (Pearson) statistics. The latter allow to establish posterior concentration rates with respect to Fisher distance rather than with respect to the Hellinger distance as it is commonplace in non-parametric Bayesian statistics.