The Annals of Statistics

Bayesian analysis of mixture models with an unknown number of components—an alternative to reversible jump methods

Matthew Stephens

Full-text: Open access

PDF File (451 KB)

Abstract

Richardson and Green present a method of performing a Bayesian analysis of data from a finite mixture distribution with an unknown number of components. Their method is a Markov Chain Monte Carlo (MCMC) approach, which makes use of the “reversible jump” methodology described by Green. We describe an alternative MCMC method which views the parameters of the model as a (marked) point process, extending methods suggested by Ripley to create a Markov birth-death process with an appropriate stationary distribution. Our method is easy to implement, even in the case of data in more than one dimension, and we illustrate it on both univariate and bivariate data. There appears to be considerable potential for applying these ideas to other contexts, as an alternative to more general reversible jump methods, and we conclude with a brief discussion of how this might be achieved.

Article information

Source
Ann. Statist., Volume 28, Number 1 (2000), 40-74.

Dates
First available in Project Euclid: 14 March 2002

Permanent link to this document
https://projecteuclid.org/euclid.aos/1016120364

Digital Object Identifier
doi:10.1214/aos/1016120364

Mathematical Reviews number (MathSciNet)
MR1762903

Zentralblatt MATH identifier
1106.62316

Subjects
Primary: 62F15: Bayesian inference

Keywords
Bayesian analysis birth-death process Markov process MCMC mixture model model choice reversible jump spatial point process

Citation

Stephens, Matthew. Bayesian analysis of mixture models with an unknown number of components—an alternative to reversible jump methods. Ann. Statist. 28 (2000), no. 1, 40--74. doi:10.1214/aos/1016120364. https://projecteuclid.org/euclid.aos/1016120364


Export citation

References

  • Anderson, E. (1935). The irises of the Gaspe Peninsula. Bulletin of the American Iris Society 59 2-5.
  • Baddeley, A. J. and van Lieshout, M. N. M. (1993). Stochastic geometry models in high-level vision. In Advances in Applied Statistics (K. V. Mardia and G. K. Kanji, eds.) 1 231-256. Carfax, Abingdon, UK.
  • Brooks, S. P. and Roberts, G. O. (1998). Convergence assessment techniques for Markov chain Monte Carlo. Statist. Comput. 8 319-335.
  • Carlin, B. P. and Chib, S. (1995). Bayesian model choice via Markov Chain Monte Carlo methods. J. Roy. Statist. Soc. Ser. B 57 473-484.
    Zentralblatt MATH: 0827.62027
  • Chib, S. (1995). Marginal likelihood from the Gibbs output. J. Amer. Statist. Assoc. 90 1313-1321.
    Mathematical Reviews (MathSciNet): MR96k:62069
    Zentralblatt MATH: 0868.62027
    Digital Object Identifier: doi:10.2307/2291521
    JSTOR: links.jstor.org
  • Cowles, M. K. and Carlin, B. P. (1996). Markov Chain Monte Carlo convergence diagnostics: a comparative review. J. Amer. Statist. Assoc. 91 883-904.
    Mathematical Reviews (MathSciNet): MR1395755
    Zentralblatt MATH: 0869.62066
    Digital Object Identifier: doi:10.2307/2291683
    JSTOR: links.jstor.org
  • Crawford, S. L. (1994). An application of the Laplace method to finite mixture distributions. J. Amer. Statist. Assoc. 89 259-267.
    Mathematical Reviews (MathSciNet): MR94k:62038
    Zentralblatt MATH: 0795.62022
    Digital Object Identifier: doi:10.2307/2291222
    JSTOR: links.jstor.org
  • Dawid, A. P. (1997). Contribution to the discussion of paper by Richardson and Green (1997). J. Roy. Statist. Soc. Ser. B 59 772-773.
  • DiCiccio, T., Kass, R., Raftery, A. and Wasserman, L. (1997). Computing Bayes factors by posterior simulation and asymptotic approximations. J. Amer. Statist. Assoc. 92 903- 915.
    Mathematical Reviews (MathSciNet): MR1482122
    Zentralblatt MATH: 1050.62520
    Digital Object Identifier: doi:10.2307/2965554
    JSTOR: links.jstor.org
  • Diebolt, J. and Robert, C. P. (1994). Estimation of finite mixture distributions through Bayesian sampling. J. Roy. Statist. Soc. Ser. B 56 363-375.
    Mathematical Reviews (MathSciNet): MR96c:62051
    JSTOR: links.jstor.org
  • Escobar, M. D. and West, M. (1995). Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90 577-588.
    Mathematical Reviews (MathSciNet): MR96d:62054
    Zentralblatt MATH: 0826.62021
    Digital Object Identifier: doi:10.2307/2291069
    JSTOR: links.jstor.org
  • Gelman, A. and Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences (with discussion). Statist. Sci. 7 457-511.
    Mathematical Reviews (MathSciNet): MR1196276
    Zentralblatt MATH: 0767.62021
  • Gelman, A. G., Carlin, J. B., Stern, H. S. and Rubin, D. B. (1995). Bayesian Data Analysis. Chapman & Hall, London.
    Mathematical Reviews (MathSciNet): MR97c:62059
  • George, E. I. and McCulloch, R. E. (1996). Stochastic search variable selection. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) Chapman & Hall, London.
    Mathematical Reviews (MathSciNet): MR1397966
  • Geyer, C. J. and Møller, J. (1994). Simulation procedures and likelihood inference for spatial point processes. Scand. J. Statist. 21 359-373.
    Mathematical Reviews (MathSciNet): MR95i:62082
  • Gilks, W. R., Richardson, S. and Spiegelhalter, D. J., eds. (1996). Markov Chain Monte Carlo in Practice. Chapman & Hall, London.
    Mathematical Reviews (MathSciNet): MR97d:62006
    Zentralblatt MATH: 0832.00018
  • Gl ¨otzl, E. (1981). Time-reversible and Gibbsian point processes I. Markovian spatial birth and death process on a general phase space. Math. Nach. 102 217-222.
  • Green, P. J. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82 711-732.
    Zentralblatt MATH: 0861.62023
    Mathematical Reviews (MathSciNet): MR1380810
    Digital Object Identifier: doi:10.1093/biomet/82.4.711
    JSTOR: links.jstor.org
  • H¨ardle, W. (1991). Smoothing techniques with implementation in S. Springer, New York.
    Mathematical Reviews (MathSciNet): MR93b:62006
  • Jennison, C. (1997). Comment on "On Bayesian analysis of mixtures with an unknown number of components," by S. Richardson and P. J. Green. J. Roy. Statist. Soc. Ser. B 59 778-779. Kelly, F. P. and Ripley, B. D. (1976) A note on Strauss's model for clustering. Biometrika 63 357-360.
    Mathematical Reviews (MathSciNet): MR1483213
    Digital Object Identifier: doi:10.1111/1467-9868.00095
    JSTOR: links.jstor.org
  • Lawson, A. B. (1996). Markov chain Monte Carlo methods for spatial cluster processes. In Computer Science and Statistics: Proceedings of the 27th Symposium on the Interface 314-319.
  • Mathieson, M. J. (1997). Ordinal models and predictive methods in pattern recognition. Ph.D. thesis, Univ. Oxford.
  • McLachlan, G. J. (1992). Discriminant Analysis and Statistical Pattern Recognition. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR94a:62091
    Zentralblatt MATH: 1108.62317
  • McLachlan, G. J. and Basford, K. E. (1988). Mixture Models: Inference and Applications to Clustering. Dekker, New York.
    Mathematical Reviews (MathSciNet): MR926484
  • Phillips, D. B. and Smith, A. F. M. (1996). Bayesian model comparison via jump diffusions. In
    Mathematical Reviews (MathSciNet): MR1397970
    Zentralblatt MATH: 0855.62018
  • Postman, M., Huchra, J. P. and Geller, M. J. (1986). Probes of large-scale structure in the Corona Borealis region. The Astronomical Journal 92 1238-1247.
  • Preston, C. J. (1976). Spatial birth-and-death processes. Bull. Inst. Internat. Statist. 46 371-391.
    Mathematical Reviews (MathSciNet): MR474532
  • Priebe, C. E. (1994). Adaptive mixtures. J. Amer. Statist. Assoc. 89 796-806.
    Mathematical Reviews (MathSciNet): MR1294725
    Zentralblatt MATH: 0825.62445
    Digital Object Identifier: doi:10.2307/2290905
    JSTOR: links.jstor.org
  • Richardson, S. and Green, P. J. (1997). On Bayesian analysis of mixtures with an unknown number of components (with discussion). J. Roy. Statist. Soc. Ser. B 59 731-792.
    Mathematical Reviews (MathSciNet): MR1483213
    Digital Object Identifier: doi:10.1111/1467-9868.00095
    JSTOR: links.jstor.org
  • Ripley, B. D. (1977). Modelling spatial patterns (with discussion). J. Roy. Statist. Soc. Ser. B 39 172-212.
    Mathematical Reviews (MathSciNet): MR58:7839
    JSTOR: links.jstor.org
  • Ripley, B. D. (1987). Stochastic Simulation. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR88b:68181
  • Robert, C. P. (1994). The Bayesian Choice: a Decision-Theoretic Motivation. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1313727
  • Robert, C. P. (1996). Mixtures of distributions: Inference and estimation. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) Chapman & Hall, London.
    Mathematical Reviews (MathSciNet): MR1397974
    Zentralblatt MATH: 0849.62013
  • Roeder, K. (1990). Density estimation with confidence sets exemplified by superclusters and voids in the galaxies. J. Amer. Statist. Assoc. 85 617-624.
    Zentralblatt MATH: 0704.62103
  • Sheather, S. J. and Jones, M. C. (1991). A reliable data-based bandwidth selection method for kernel density estimation. J. Roy. Statist. Soc. Ser. B 53 683-690.
    Mathematical Reviews (MathSciNet): MR1125725
    JSTOR: links.jstor.org
  • Stephens, D. A. and Fisch, R. D. (1998). Bayesian analysis of quantitative trait locus data using reversible jump Markov chain Monte Carlo. Biometrics 54 1334-1367.
  • Stephens, M. (1997). Bayesian Methods for Mixtures of Normal Distributions. Ph.D. thesis, Univ. Oxford. Available from www.stats.ox.ac.uk/ stephens.
    URL: Link to item
  • Stoyan, D., Kendall, W. S. and Mecke, J. (1987). Stochastic Geometry and Its Applications, 1st ed. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR895588
  • Tierney, L. (1996). Introduction to general state-space Markov chain theory. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) 59-74. Chapman & Hall, London.
    Mathematical Reviews (MathSciNet): MR1397968
    Zentralblatt MATH: 0849.60072
  • Titterington, D. M., Smith, A. F. M. and Makov, U. E. (1985). Statistical Analysis of Finite Mixture Distributions. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR87j:62033
    Zentralblatt MATH: 0646.62013
  • Venables, W. N. and Ripley, B. D. (1994). Modern Applied Statistics with S-Plus. Springer, New York.
    Mathematical Reviews (MathSciNet): MR97c:62003
  • Venables, W. N. and Ripley, B. D. (1997). Modern Applied Statistics with S-Plus, 2nd ed. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1337030
  • West, M. (1993). Approximating posterior distributions by mixtures. J. Roy. Statist. Soc. Ser. B 55 409-422.
    Zentralblatt MATH: 0800.62221
    Mathematical Reviews (MathSciNet): MR1224405
    JSTOR: links.jstor.org
  • Wilson, S. R. (1982). Sound and exploratory data analysis. In COMPSTAT 1982. Proceedings in Computational Statistics (H. Caussinus, P. Ettinger and R. Tamassone, eds.) 447-450. Physica, Vienna.

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more