The Annals of Applied Statistics

Bayesian variable selection using cost-adjusted BIC, with application to cost-effective measurement of quality of health care

D. Fouskakis, I. Ntzoufras, and D. Draper

Full-text: Open access


In the field of quality of health care measurement, one approach to assessing patient sickness at admission involves a logistic regression of mortality within 30 days of admission on a fairly large number of sickness indicators (on the order of 100) to construct a sickness scale, employing classical variable selection methods to find an “optimal” subset of 10–20 indicators. Such “benefit-only” methods ignore the considerable differences among the sickness indicators in cost of data collection, an issue that is crucial when admission sickness is used to drive programs (now implemented or under consideration in several countries, including the U.S. and U.K.) that attempt to identify substandard hospitals by comparing observed and expected mortality rates (given admission sickness). When both data-collection cost and accuracy of prediction of 30-day mortality are considered, a large variable-selection problem arises in which costly variables that do not predict well enough should be omitted from the final scale.

In this paper (a) we develop a method for solving this problem based on posterior model odds, arising from a prior distribution that (1) accounts for the cost of each variable and (2) results in a set of posterior model probabilities that corresponds to a generalized cost-adjusted version of the Bayesian information criterion (BIC), and (b) we compare this method with a decision-theoretic cost-benefit approach based on maximizing expected utility. We use reversible-jump Markov chain Monte Carlo (RJMCMC) methods to search the model space, and we check the stability of our findings with two variants of the MCMC model composition (MC3) algorithm. We find substantial agreement between the decision-theoretic and cost-adjusted-BIC methods; the latter provides a principled approach to performing a cost-benefit trade-off that avoids ambiguities in identification of an appropriate utility structure. Our cost-benefit approach results in a set of models with a noticeable reduction in cost and dimensionality, and only a minor decrease in predictive performance, when compared with models arising from benefit-only analyses.

Article information

Ann. Appl. Stat., Volume 3, Number 2 (2009), 663-690.

First available in Project Euclid: 22 June 2009

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Input-output analysis quality of health care sickness at hospital admission cost-benefit analysis Laplace approximation reversible-jump Markov chain Monte Carlo (RJMCMC) methods MCMC model composition (MC^3) Bayesian information criterion (BIC) cost-adjusted BIC


Fouskakis, D.; Ntzoufras, I.; Draper, D. Bayesian variable selection using cost-adjusted BIC, with application to cost-effective measurement of quality of health care. Ann. Appl. Stat. 3 (2009), no. 2, 663--690. doi:10.1214/08-AOAS207.

Export citation


  • Barbieri, M. D. and Berger, J. O. (2004). Optimal predictive model selection. Ann. Statist. 32 870–897.
  • Bartlett, M. S. (1957). Comment on D. V. Lindley’s statistical paradox. Biometrika 44 533–534.
  • Bernardo, J. M. and Smith, A. F. M. (1994). Bayesian Theory. Wiley, New York.
  • Brown, P. J., Vannucci, M. and Fearn, T. (1998). Multivariate Bayesian variable selection and prediction. J. Roy. Statist. Soc. Ser. B 60 627–641.
  • CalNOC (2008). The California nursing outcomes coalition database project. Available at
  • Chen, M. H., Ibrahim, J. G. and Shao, Q. M. (2000). Power prior distributions for generalized linear models. J. Statist. Plann. Inference 84 121–137.
  • Chipman, H., George, E. I. and McCulloch, R. E. (2001). The practical implementation of Bayesian model selection (with discussion). In Model Selection. IMS Lecture Notes Monogr. Ser. 38 67–134. Institute of Mathematical Statistics, Beachwood, OH.
  • CMS (2008). Centers for Medicare & Medicaid services: Medicare information resource. Available at
  • Dellaportas, P., Forster, J. J. and Ntzoufras, I. (2002). On Bayesian model and variable selection using MCMC. Statist. Comput. 12 27–36.
  • Dempster, A. P. (1974). The direct use of likelihood for significance testing. In Proceedings of a Conference on Foundational Questions in Statistical Inference (O. Barndorff-Nielsen, P. Blaesild and G. Sihon, eds.) 335–352. Univ. Aarhus, Aarhus. [Reprinted: Statist. Comput. 7 (1997) 247–252].
  • Donabedian, A. and Bashshur, R. (2002). An Introduction to Quality Assurance in Health Care. Oxford Univ. Press, Oxford.
  • Draper, D. (1995). Inference and hierarchical modeling in the social sciences (with discussion). Journal of Educational and Behavioral Statistics 20 115–147, 233–239.
  • Draper, D. and Fouskakis, D. (2000). A case study of stochastic optimization in health policy: Problem formulation and preliminary results. Journal of Global Optimization 18 399–416.
  • Draper, D. and Krnjajić, M. (2009). Bayesian model specification. Unpublished manuscript.
  • Fouskakis, D. (2001). Stochastic optimisation methods for cost-effective quality assessment in health. Ph.D. dissertation, Dept. Mathematical Sciences, Univ. Bath, UK. Available at
  • Fouskakis, D. and Draper, D. (2002). Stochastic optimization: A review. Int. Statist. Rev. 70 315–349.
  • Fouskakis, D. and Draper, D. (2008). Comparing stochastic optimization methods for variable selection in binary outcome prediction, with application to health policy. J. Amer. Statist. Assoc. 103 1367–1381.
  • Fouskakis, D., Ntzoufras, I. and Draper, D. (2009a). Supplement to “Bayesian variable selection using cost-adjusted BIC with application to cost-effective measurement of quality health care.” DOI:10.1214/08-AOAS207SUPP.
  • Fouskakis, D., Ntzoufras, I. and Draper, D. (2009b). Population-based reversible jump MCMC for Bayesian variable selection and evaluation under a cost constraint. J. Roy. Statist. Soc. Ser. C 58 383–403.
  • Geisser, S. and Eddy, W. F. (1979). A predictive approach to model selection. J. Amer. Statist. Assoc. 74 153–160.
  • Gelfand, A. E. (1996). Model determination using sampling-based methods. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) 145–162. Chapman and Hall, London.
  • Gelfand, A. E., Dey, D. K. and Chang, H. (1992). Model determination using predictive distributions, with implementation via sampling-based methods (with discussion). In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 147–167. Oxford Univ. Press, Oxford.
  • George, E. I. and McCulloch, R. E. (1993). Variable selection via Gibbs sampling. J. Amer. Statist. Assoc. 88 881–889.
  • Goldstein, H. and Spiegelhalter, D. J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance (with discussion). J. Roy. Statist. Soc. Ser. A 159 385–444.
  • Green, P. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82 711–732.
  • Han, C. and Carlin, B. (2001). MCMC methods for computing Bayes factors: A comparative review. J. Amer. Statist. Assoc. 96 1122–1132.
  • Hoeting, J. A., Madigan, D., Raftery, A. E. and Volinski, C. T. (1999). Bayesian model averaging: A tutorial. Statist. Sci. 14 382–417.
  • Kahn, K., Rogers, W., Rubenstein, L., Sherwood, M., Reinisch, E., Keeler, E., Draper, D., Kosecoff, J. and Brook, R. (1990a). Measuring quality of care with explicit process criteria before and after implementation of the DRG-based Prospective Payment System. J. Amer. Med. Assoc. 264 1969–1973 (with editorial comment, 1995–1997).
  • Kahn, K., Rubenstein, L., Draper, D., Kosecoff, J., Rogers, W., Keeler, E. and Brook, R. (1990b). The effects of the DRG-based Prospective Payment System on quality of care for hospitalized Medicare patients: An introduction to the series. J. Amer. Med. Assoc. 264 1953–1955 (with editorial comment, 1995–1997).
  • Kass, R. E. and Raftery, A. E. (1995). Bayes factors. J. Amer. Statist. Assoc. 90 773–795.
  • Kass, R. E. and Wasserman, L. (1996). The selection of prior distributions by formal rules. J. Amer. Statist. Assoc. 91 1343–1370.
  • Keeler, E., Kahn, K., Draper, D., Sherwood, M., Rubenstein, L., Reinisch, E., Kosecoff, J. and Brook, R. (1990). Changes in sickness at admission following the introduction of the Prospective Payment System. J. Amer. Med. Assoc. 264 1962–1968.
  • Kuo, L. and Mallick, B. (1998). Variable selection for regression models. Sankhyā Ser. B 60 65–81.
  • Lindley, D. V. (1957). A statistical paradox. Biometrika 44 187–192.
  • Lindley, D. V. (1968). The choice of variables in multiple regression (with discussion). J. Roy. Statist. Soc. Ser. B 30 31–66.
  • Lopes, H. F. (2002). Bayesian model selection. Technical report, Dept. Métodos Estatísticos, Univ. Federal do Rio de Janeiro, Brazil.
  • Madigan, D. and York, J. (1995). Bayesian graphical models for discrete data. Int. Statist. Rev. 63 215–232.
  • McCullagh, P. and Nelder, J. A. (1983). Generalized Linear Models. Chapman and Hall, London.
  • NDNQI (2008). National database of nursing quality indicators. Available at www.nursingquality. org.
  • Ntzoufras, I. (1999). Aspects of Bayesian model and variable selection using MCMC. Ph.D. thesis, Department of Statistics, Athens University of Economics and Business. Available at
  • Ntzoufras, I., Dellaportas, P. and Forster, J. J. (2003). Bayesian variable and link determination for generalized linear models. J. Statist. Plann. Inference 111 165–180.
  • Ohlssen, D. I., Sharples, L. D. and Spiegelhalter, D. J. (2007). A hierarchical modelling framework for identifying unusual performance in health care providers. J. Roy. Statist. Soc. Ser. A 170 865–890.
  • Raftery, A. E. (1995). Bayesian model selection in social research. In Sociological Methodology 1995 (P. V. Marsden, ed.) 25 111–196. Blackwell, Oxford.
  • Raftery, A. E. (1996). Approximate Bayes factors and accounting for model uncertainty in generalized linear models. Biometrika 83 251–266.
  • Robert, C. P. (1993). A note on the Jeffreys–Lindley paradox. Statist. Sinica 3 601–608.
  • Schuster, M. A., McGlynn, E. A. and Brook, R. H. (2005). How good is the quality of health care in the United States? Milbank Quarterly 83 843–895.
  • Schwarz, G. (1978). Estimating the dimension of a model. Ann. Statist. 6 461–464.
  • Shafer, J. (1982). Lindley’s paradox (with discussion). J. Amer. Statist. Assoc. 77 325–334.
  • Sinharay, S. and Stern, H. S. (2002). On the sensitivity of Bayes factors to the prior distributions. Amer. Statist. 56 196–201.
  • Spiegelhalter, D. J., Best, N., Carlin, B. and van der Linde, A. (2002). Bayesian measures of model complexity and fit (with discussion). J. Roy. Statist. Soc. Ser. B 64 583–639.
  • Spiegelhalter, D. J., Thomas, A., Best, N. and Gilks, W. (1996). BUGS 0.5: Bayesian inference using Gibbs sampling. Available at
  • Tierney, L. and Kadane, J. B. (1986). Accurate approximations for posterior moments and marginal densities. J. Amer. Statist. Assoc. 81 82–86.
  • Zhang, M., Strawderman, R. L., Cowen, M. E. and Wells, M. T. (2006). Bayesian inference for a two-part hierarchical model: An application to profiling providers in managed health care. J. Amer. Statist. Assoc. 101 934–945.

Supplemental materials