Brazilian Journal of Probability and Statistics

Keeping the balance—Bridge sampling for marginal likelihood estimation in finite mixture, mixture of experts and Markov mixture models

Sylvia Frühwirth-Schnatter

Abstract

Finite mixture models and their extensions to Markov mixture and mixture of experts models are very popular in analysing data of various kind. A challenge for these models is choosing the number of components based on marginal likelihoods. The present paper suggests two innovative, generic bridge sampling estimators of the marginal likelihood that are based on constructing balanced importance densities from the conditional densities arising during Gibbs sampling. The full permutation bridge sampling estimator is derived from considering all possible permutations of the mixture labels for a subset of these densities. For the double random permutation bridge sampling estimator, two levels of random permutations are applied, first to permute the labels of the MCMC draws and second to randomly permute the labels of the conditional densities arising during Gibbs sampling. Various applications show very good performance of these estimators in comparison to importance and to reciprocal importance sampling estimators derived from the same importance densities.

Article information

Source
Braz. J. Probab. Stat., Volume 33, Number 4 (2019), 706-733.

Dates
Accepted: April 2019
First available in Project Euclid: 26 August 2019

https://projecteuclid.org/euclid.bjps/1566806429

Digital Object Identifier
doi:10.1214/19-BJPS446

Mathematical Reviews number (MathSciNet)
MR3996313

Citation

Frühwirth-Schnatter, Sylvia. Keeping the balance—Bridge sampling for marginal likelihood estimation in finite mixture, mixture of experts and Markov mixture models. Braz. J. Probab. Stat. 33 (2019), no. 4, 706--733. doi:10.1214/19-BJPS446. https://projecteuclid.org/euclid.bjps/1566806429

References

• Aßmann, C. and Boysen-Hogrefe, J. (2011). A Bayesian approach to model-based clustering for binary panel probit models. Computational Statistics & Data Analysis 55, 261–279.
• Berger, J. O. and Jefferys, W. H. (1992). Sharpening Ockham’s razor on a Bayesian strop. American Statistician 80, 64–72.
• Berkhof, J., van Mechelen, I. and Gelman, A. (2003). A Bayesian approach to the selection and testing of mixture models. Statistica Sinica 13, 423–442.
• Celeux, G., Frühwirth-Schnatter, S. and Robert, C. P. (2019). Model selection for mixture models—perspectives and strategies. In Handbook of Mixture Analysis (S. Frühwirth-Schnatter, G. Celeux and C. P. Robert, eds.) 117–154. Boca Raton, FL: CRC Press.
• Chib, S. (1995). Marginal likelihood from the Gibbs output. Journal of the American Statistical Association 90, 1313–1321.
• DiCiccio, T. J., Kass, R. E., Raftery, A. and Wasserman, L. (1997). Computing Bayes factors by combining simulation and asymptotic approximations. Journal of the American Statistical Association 92, 903–915.
• Diebolt, J. and Robert, C. P. (1994). Estimation of finite mixture distributions through Bayesian sampling. Journal of the Royal Statistical Society, Series B 56, 363–375.
• Diggle, P. J., Heagerty, P., Liang, K.-Y. and Zeger, S. L. (2002). Analysis of Longitudinal Data, 2nd ed. Oxford: Oxford University Press.
• Escobar, M. D. and West, M. (1998). Computing nonparametric hierarchical models. In Practical Nonparametric and Semiparametric Bayesian Statistics (D. Dey, P. Müller and D. Sinha, eds.), Lecture Notes in Statistics 133, 1–22. Berlin: Springer.
• Frühwirth-Schnatter, S. (1995). Bayesian model discrimination and Bayes factors for linear Gaussian state space models. Journal of the Royal Statistical Society, Series B 57, 237–246.
• Frühwirth-Schnatter, S. (2001). Markov chain Monte Carlo estimation of classical and dynamic switching and mixture models. Journal of the American Statistical Association 96, 194–209.
• Frühwirth-Schnatter, S. (2004). Estimating marginal likelihoods for mixture and Markov switching models using bridge sampling techniques. Econometrics Journal 7, 143–167.
• Frühwirth-Schnatter, S. (2006). Finite Mixture and Markov Switching Models. New York: Springer.
• Frühwirth-Schnatter, S. (2011). Panel data analysis—A survey on model-based clustering of time series. Advances in Data Analysis and Classification 5, 251–280.
• Frühwirth-Schnatter, S. (2019). Applied Bayesian Mixture Modelling. Implementations in MATLAB using the package bayesf (Version 4.0). Available at https://www.wu.ac.at/statmath/faculty-staff/faculty/sfruehwirthschnatter/.
• Frühwirth-Schnatter, S., Celeux, G. and Robert, C. P., eds. (2019). Handbook of Mixture Analysis. Boca Raton, FL: CRC Press.
• Frühwirth-Schnatter, S. and Frühwirth, R. (2010). Data augmentation and MCMC for binary and multinomial logit models. In Statistical Modelling and Regression Structures—Festschrift in Honour of Ludwig Fahrmeir (T. Kneib and G. Tutz, eds.) 111–132. Heidelberg: Physica-Verlag.
• Frühwirth-Schnatter, S., Frühwirth, R., Held, L. and Rue, H. (2009). Improved auxiliary mixture sampling for hierarchical models of non-Gaussian data. Statistics and Computing 19, 479–492.
• Frühwirth-Schnatter, S. and Kaufmann, S. (2008). Model-based clustering of multiple time series. Journal of Business & Economic Statistics 26, 78–89.
• Frühwirth-Schnatter, S. and Malsiner-Walli, G. (2019). From here to infinity: Sparse finite versus Dirichlet process mixtures in model-based clustering. Advances in Data Analysis & Classification 13, 33–64.
• Frühwirth-Schnatter, S., Pamminger, C., Weber, A. and Winter-Ebmer, R. (2012). Labor market entry and earnings dynamics: Bayesian inference using mixtures-of-experts Markov chain clustering. Journal of Applied Econometrics 27, 1116–1137.
• Frühwirth-Schnatter, S., Pittner, S., Weber, A. and Winter-Ebmer, R. (2018). Analysing plant closure effects using time-varying mixture-of-experts Markov chain clustering. Annals of Applied Statistics 12, 1786–1830.
• Frühwirth-Schnatter, S. and Pyne, S. (2010). Bayesian inference for finite mixtures of univariate and multivariate skew normal and skew-$t$ distributions. Biostatistics 11, 317–336.
• Gelfand, A. E. and Dey, D. K. (1994). Bayesian model choice: Asymptotics and exact calculations. Journal of the Royal Statistical Society, Series B 56, 501–514.
• Gelfand, A. E. and Smith, A. F. M. (1990). Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association 85, 398–409.
• Geweke, J. (1989). Bayesian inference in econometric models using Monte Carlo integration. Econometrica 57, 1317–1339.
• Gormley, I. C. and Frühwirth-Schnatter, S. (2019). Mixture of experts models. In Handbook of Mixture Analysis (S. Frühwirth-Schnatter, G. Celeux and C. P. Robert, eds.) 271–307. Boca Raton, FL: CRC Press.
• Lee, J. E. and Robert, C. P. (2016). Importance sampling schemes for evidence approximation in mixture models. Bayesian Analysis 11, 573–597.
• Lee, K., Marin, J.-M., Mengersen, K. and Robert, C. (2009). Bayesian inference on mixtures of distributions. In Perspectives in Mathematical Sciences I: Probability and Statistics (N. N. Sastry, M. Delampady and B. Rajeev, eds.) 165–202. Singapore: World Scientific.
• Leroux, B. G. and Puterman, M. L. (1992). Maximum-penalized-likelihood estimation for independent and Markov-dependent mixture models. Biometrics 48, 545–558.
• Malsiner Walli, G., Frühwirth-Schnatter, S. and Grün, B. (2016). Model-based clustering based on sparse finite Gaussian mixtures. Statistics and Computing 26, 303–324.
• Meng, X.-L. and Schilling, S. (1996). Fitting full-information item factor models and an empirical investigation of bridge sampling. Journal of the American Statistical Association 91, 1254–1267.
• Meng, X.-L. and Wong, W. H. (1996). Simulating ratios of normalizing constants via a simple identity: A theoretical exploration. Statistica Sinica 6, 831–860.
• Polson, N. G., Scott, J. G. and Windle, J. (2013). Bayesian inference for logistic models using Pólya–Gamma latent variables. Journal of the American Statistical Association 108, 1339–1349.
• Richardson, S. and Green, P. J. (1997). On Bayesian analysis of mixtures with an unknown number of components. Journal of the Royal Statistical Society, Series B 59, 731–792.
• Robert, C. P. and Casella, G. (1999). Monte Carlo Statistical Methods. Springer Series in Statistics. New York/Berlin/Heidelberg: Springer.
• Rousseau, J., Grazian, C. and Lee, J. E. (2019). Bayesian mixture models: Theory and methods. In Handbook of Mixture Analysis (S. Frühwirth-Schnatter, G. Celeux and C. P. Robert, eds.) 53–72. Boca Raton, FL: CRC Press.
• Schwerdt, G., Ichino, A., Ruf, O., Winter-Ebmer, R. and Zweimüller, J. (2010). Does the color of the collar matter? Employment and earnings after plant closure. Economics Letters 108, 137–140.
• Zweimüller, J., Winter-Ebmer, R., Lalive, R., Kuhn, A., Wuellrich, J.-P., Ruf, O. and Büchi, S. (2009). The Austrian Social Security Database (ASSD). Working Paper 0903, NRN: The Austrian Center for Labor Economics and the Analysis of the Welfare State, Linz, Austria.