Bayesian Analysis

Bayesian auxiliary variable models for binary and multinomial regression

Leonhard Held and Chris C. Holmes

Full-text: Open access


In this paper we discuss auxiliary variable approaches to Bayesian binary and multinomial regression. These approaches are ideally suited to automated Markov chain Monte Carlo simulation. In the first part we describe a simple technique using joint updating that improves the performance of the conventional probit regression algorithm. In the second part we discuss auxiliary variable methods for inference in Bayesian logistic regression, including covariate set uncertainty. Finally, we show how the logistic method is easily extended to multinomial regression models. All of the algorithms are fully automatic with no user set parameters and no necessary Metropolis-Hastings accept/reject steps.

Article information

Bayesian Anal., Volume 1, Number 1 (2006), 145-168.

First available in Project Euclid: 22 June 2012

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Auxiliary variables Bayesian binary and multinomial regression Markov chain Monte Carlo Model averaging Scale mixture of normals Variable selection


Holmes, Chris C.; Held, Leonhard. Bayesian auxiliary variable models for binary and multinomial regression. Bayesian Anal. 1 (2006), no. 1, 145--168. doi:10.1214/06-BA105.

Export citation


  • Albert, J. and Chib, S. (1993). "Bayesian analysis of binary and polychotomous response data." Journal of the American Statistical Association, 88:669–679.
  • –- (2001). "Sequential ordinal modeling with applications to survival data." Biometrics, 57:829–836.
  • Andrews, D. and Mallows, C. (1974). "Scale mixtures of normal distributions." J. R. Statist. Soc. B, 36:99–102.
  • Brooks, S., Giudici, P., and Roberts, G. O. (2003). "Efficient construction of reversible jump MCMC proposal distributions (with discussion)." J. R. Statist. Soc. B, 65:3–55.
  • Bull, S. (1994). Analysis of attitudes toward workplace smoking restrictions. Wiley. Lange, N., Ryan, L., Billard, L., Brillinger, D., Conquest, L. and Greenhouse, J. (eds.).
  • Chen, M.-H. and Dey, D. (1998). "Bayesian modeling of correlated binary responses via scale mixture of multivariate normal link functions." Sankyā: The Indian Journal of Statistics, 60:322–343.
  • Clyde, M. (1999). "Bayesian model averaging and model search stratergies (with discussion)." In Bernardo, J., Berger, J., Dawid, A., and Smith, A. (eds.), Bayesian Statisics 6. Oxford: Clarendon Press.
  • Dellaportas, P. and Smith, A. (1993). "Bayesian inference for generalized linear and proportional hazard models via Gibbs sampling." Appl. Statist., 42:443–459.
  • Denison, D., Holmes, C., Mallick, B., and Smith, A. (eds.) (2002). Bayesian methods for nonlinear classification and regression.. Chichester: Wiley.
  • Devroye, L. (1986). Non-Uniform Random Variate Generation. New York: Springer.
  • Dey, D., Ghosh, S., and Mallick, B. (eds.) (1999). Generalized Linear Models: A Bayesian Perspective. New York: Marcel Dekker.
  • Gamerman, D. (1997). "Efficient sampling from the posterior distribution in generalized linear mixed models." Statistics and Computing, 7:57–68.
  • Gelfand, A. and Smith, A. (1990). "Sampling-based approaches to calculating marginal densities." Journal of the American Statistical Association, 85:398–409.
  • Geyer, C. (1992). "Practical Markov chain Monte Carlo." Statistical Science, 7:473–511.
  • Green, P. (1995). "Markov chain Monte Carlo computation and Bayesian model determination." Biometrika, 82:711–732.
  • Held, L. (2004). "Simultaneous posterior probability statements from Monte Carlo output." Journal of Computational and Graphical Statistics, 13:20–35.
  • Henderson, H. and Searle, S. (1981). "On deriving the inverse of a sum of matrices." SIAM Rev., 23:53–60.
  • Kass, R., Carlin, B., A., G., and Neal, R. (1998). "Markov Chain Monte Carlo in Practice: A Roundtable Discussion." The American Statistician, 52:93–100.
  • McCullagh, P. and Nelder, J. (1989). Generalised Linear Models. London: Chapman and Hall, 2nd edition.
  • Michie, D., Spiegelhalter, D., and Taylor, C. (1994). Machine Learning, Neural and Statistical Classification. Ellis Horwood.
  • Ripley, B. (1996). Pattern Recognition and Neural Networks. Cambridge University Press.
  • Robert, C. (1995). "Simulation of truncated normal variables." Statistics and Computing, 5:121–125.

See also

  • Related item: Ralf van der Lans. Bayesian estimation of the multinomial logit model: a comment on Holmes and Held, "Bayesian auxiliary variable models for binary and multinomial regression" . Bayesian Anal., Vol. 6, Iss.2 (2011), 353-355.