Statistical Science

Compatibility of Prior Specifications Across Linear Models

Guido Consonni and Piero Veronese

Full-text: Open access

Abstract

Bayesian model comparison requires the specification of a prior distribution on the parameter space of each candidate model. In this connection two concerns arise: on the one hand the elicitation task rapidly becomes prohibitive as the number of models increases; on the other hand numerous prior specifications can only exacerbate the well-known sensitivity to prior assignments, thus producing less dependable conclusions. Within the subjective framework, both difficulties can be counteracted by linking priors across models in order to achieve simplification and compatibility; we discuss links with related objective approaches. Given an encompassing, or full, model together with a prior on its parameter space, we review and summarize a few procedures for deriving priors under a submodel, namely marginalization, conditioning, and Kullback–Leibler projection. These techniques are illustrated and discussed with reference to variable selection in linear models adopting a conventional g-prior; comparisons with existing standard approaches are provided. Finally, the relative merits of each procedure are evaluated through simulated and real data sets.

Article information

Source
Statist. Sci., Volume 23, Number 3 (2008), 332-353.

Dates
First available in Project Euclid: 28 January 2009

Permanent link to this document
https://projecteuclid.org/euclid.ss/1233153062

Digital Object Identifier
doi:10.1214/08-STS258

Mathematical Reviews number (MathSciNet)
MR2483907

Zentralblatt MATH identifier
1329.62331

Keywords
Bayes factor compatible prior conjugate prior g-prior hypothesis testing Kullback–Leibler projection nested model variable selection

Citation

Consonni, Guido; Veronese, Piero. Compatibility of Prior Specifications Across Linear Models. Statist. Sci. 23 (2008), no. 3, 332--353. doi:10.1214/08-STS258. https://projecteuclid.org/euclid.ss/1233153062


Export citation

References

  • Barbieri, M. M. and Berger, J. O. (2004). Optimal predictive model selection. Ann. Statist. 32 870–897.
  • Bayarri, M. J. and García-Donato, G. (2007). Extending conventional priors for testing general hypotheses in linear models. Biometrika 94 135–152.
  • Berger, J. O. and Pericchi, L. R. (1996a). The intrinsic Bayes factor for model selection and prediction. J. Amer. Statist. Assoc. 91 109–122.
  • Berger, J. O. and Pericchi, L. R. (1996b). The intrinsic Bayes factor for linear models. In Bayesian Statistics 5 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 25–44. Oxford Univ. Press.
  • Berger, J. O. and Pericchi, L. (2001). Objective Bayesian methods for model selection: Introduction and comparison (with discussion). In Model Selection (P. Lahiri, ed.) 135–207. Inst. Math. Statist., Beachwood, OH.
  • Berger, J. O. and Pericchi, L. (2004). Training samples in objective Bayesian model selection. Ann. Statist. 32 841–869.
  • Bernardo, J. M. and Rueda, R. (2002). Bayesian hypothesis testing: A reference approach. Internat. Statist. Rev. 70 351–372.
  • Casella, G. and Moreno, E. (2005). Intrinsic meta-analysis of contingency tables. Stat. Med. 24 583–604.
  • Casella, G. and Moreno, E. (2006). Objective Bayesian variable selection. J. Amer. Statist. Assoc. 101 157–167.
  • Casella, G. and Moreno, E. (2007). Assessing robustness of intrinsic tests of independence in two-way contingency tables. Technical report, Dept. Statistics, Univ. Florida.
  • Celeux, G., Marin, J. M. and Robert, C. P. (2006). Sélection bayésienne de variables en régression linéaire. J. Soc. Française de Statistique 147 59–79.
  • Clyde, M., DeSimone, H. and Parmigiani, G. (1996). Prediction via orthogonalized model mixing. J. Amer. Statist. Assoc. 91 1197–1208.
  • Clyde, M. and George, E. I. (2004). Model uncertainty. Statist. Sci. 19 81–94.
  • Consonni, G., Gutiérrez-Peña, E. and Veronese, P. (2007). Compatible priors for Bayesian model comparison with an application to the Hardy–Weinberg equilibrium model. Test. To appear. DOI 10.1007/s11749-007-0057-7.
  • Consonni, G. and La Rocca, L. (2008). Tests based on intrinsic priors for the equality of two correlated proportions. J. Amer. Statist. Assoc. To appear.
  • Cripps, E., Carter, C. and Kohn, R. (2005). Variable selection and covariance selection in multivariate regression models. In Handbook of Statistics 25. Bayesian Thinking Modeling and Computation (D. K. Dey and C. R. Rao, eds.) 519–552. North-Holland, New York.
  • Davison, A. C. (2003). Statistical Models. Cambridge Univ. Press, Cambridge.
  • Dawid, A. P. and Lauritzen, S. L. (2001). Compatible prior distributions. In Bayesian Methods With Applications to Science, Policy and Official Statistics (E. George, ed.) 109–118. Monographs of Official Statistics, Luxembourg.
  • Dey, D. K. and Rao, C. R., eds. (2005). Handbook of Statistics 25. Bayesian Thinking Modeling and Computation. North-Holland, New York.
  • Dickey, J. M. (1971). The weighted likelihood ratio, linear hypotheses on normal location parameters. Ann. Math. Statist. 42 204–223.
  • Draper, N. R. and Smith, H. (1981). Applied Regression Analysis, 2nd ed. Wiley, New York.
  • Dupuis, J. A. and Robert, C. P. (2003). Variable selection in qualitative models via an entropic explanatory power. J. Statist. Plann. Inference 111 77–94.
  • Fernández, C., Ley, E. and Steel, M. F. J. (2001). Benchmark priors for Bayesian model averaging. J. Econometrics 100 381–427.
  • Freedman, D. A. (1983). A note on screening regression equations. Amer. Statist. 37 152–155.
  • Garthwaite, P. H. and Dickey, J. M. (1996). Quantifying and using expert opinion for variable-selection problems in regression. Chemometrics and Intelligent Laboratory Systems 35 1–26.
  • Gelfand, A. E. and Ghosh, S. K. (1998). Model choice: A minimum posterior predictive loss approach. Biometrika 85 1–11.
  • George, E. I. and McCulloch, R. E. (1993). Variable selection via Gibbs sampling. J. Amer. Statist. Assoc. 88 881–889.
  • George, E. I. (1999). Bayesian model selection. In Encyclopedia of Statistical Sciences, Update 3 (S. Kotz, C. Read and D. Banks, eds.) 39–46. Wiley, New York.
  • George, E. I. (2000). The variable selection problem. J. Amer. Statist. Assoc. 95 1304–1308.
  • George, E. I. and Foster, D. P. (2000). Calibration and empirical Bayes variable selection. Biometrika 87 731–747.
  • George, E. I. and McCulloch, R. E. (1997). Approaches for Bayesian variable selection. Statist. Sinica 7 339–374.
  • Goutis, C. and Robert, C. P. (1998). Model choice in generalised linear models: A Bayesian approach via Kullback–Leibler projections. Biometrika 85 29–37.
  • Ibrahim, J. G. (1997). On properties of predictive priors in linear models. Amer. Statist. 51 333–337.
  • Laud, P. W. and Ibrahim, J. G. (1995). Predictive model selection. J. Roy. Statist. Soc. Ser. B 57 247–262.
  • Laud, P. W. and Ibrahim, J. G. (1996). Predictive specification of prior model probabilities in variable selection. Biometrika 83 267–274.
  • Liang, R. F., Paulo, R., Molina, G., Clyde, M. and Berger, J. O. (2008). Mixtures of g-priors for Bayesian variable selection. J. Amer. Statist. Assoc. 103 410–423.
  • O’Hagan, A. and Forster, J. (2004). Kendall’s Advanced Theory of Statistics 2b. Bayesian Inference, 2nd ed. Edward Arnold, London.
  • Marriott, J. M., Spencer, N. M. and Pettitt, N. (2001). A Bayesian approach to selecting covariates for prediction. Scand. J. Statist. 28 87–97.
  • McCulloch, R. E. and Rossi, P. E. (1992). Bayes factor for nonlinear hypotheses and likelihood distributions. Biometrika 79 663–676.
  • Moreno, E. and Giron, E. J. (2007). Comparison of Bayesian objective procedures for variable selection in linear regression. Test. To appear. DOI 10.1007/s11749-006-0039-1.
  • Morris, C. M. (1987). Comment on: “Reconciling Bayesian and frequentist evidence in the one-sided testing problem,” by G. Casella and R. L. Berger and “Testing a point null hypothesis: The irreconcilability of P values and evidence,” by J. O. Berger and T. Sellke. J. Amer. Statist. Assoc. 82 131–135.
  • Muirhead, R. J. (1982). Aspects of Multivariate Statistical Theory. Wiley, New York.
  • Neal, R. (2001). Transferring prior information between models using imaginary data. Technical Report No. 0108, Dept. Statistics, Univ. Toronto.
  • Nott, D. J. and Green, P. J. (2004). Bayesian variable selection and the Swendsen–Wang algorithm. J. Comput. Graph. Statist. 13 141–157.
  • Pérez, J. M. and Berger, J. O. (2002). Expected-posterior prior distributions for model selection. Biometrika 89 491–511.
  • Pericchi, L. R. (2005). Model selection and hypothesis testing based on objective probabilities and Bayes factors. In Handbook of Statistics 25. Bayesian Thinking Modeling and Computation (D. K. Dey and C. R. Rao, eds.) 115–149. North-Holland, New York.
  • Poirier, D. J. (1985). Bayesian hypothesis testing in linear models with continuously induced conjugate priors across hypotheses. In Bayesian Statistics 2 (J. M. Bernardo, M. H. DeGroot, D. V. Lindley and A. F. M. Smith, eds.) 711–722. North-Holland, Amsterdam.
  • Raftery, A. E., Madigan, D. and Hoeting, J. A. (1997). Bayesian model averaging for linear regression models. J. Amer. Statist. Assoc. 92 179–191.
  • Rao, R. and Toutemburg, H. (1999). Linear Models: Least Squares and Alternatives. Springer, New York.
  • Robert, C. P. (2001). The Bayesian Choice, 2nd ed. Springer, New York.
  • Roverato, A. and Consonni, G. (2004). Compatible prior distributions for directed acyclic graph models. J. Roy. Statist. Soc. Ser. B 66 47–61.
  • Searle, S. R. (1982). Matrix Algebra Useful for Statistics. Wiley, Chichester.
  • Smith, M. and Kohn, R. (1996). Nonparametric regression using Bayesian variable selection. J. Econometrics 75 317–343.
  • Viele, K. and Srinivasan, C. (2000). Parsimonious estimation of multiplicative interaction in analysis of variance using Kullback–Leibler information. J. Statist. Plann. Inference 84 201–219.
  • Yuan, M. and Lin, Y. (2005). Efficient empirical Bayes variable selection and estimation in linear models. J. Amer. Statist. Assoc. 100 1215–1225.
  • Whittaker, J. (1990). Graphical Models in Applied Multivariate Statistics. Wiley, Chichester.
  • Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with g-prior distributions. In Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti (P. K. Goel and A. Zellner, eds.) 233–243. North-Holland, Amsterdam.
  • Zellner, A. and Siow, A. (1980). Posterior odds ratio for selected regression hypotheses. In Bayesian Statistics 1 (J. M. Bernardo, M. H. DeGroot, D. V. Lindley and A. F. M. Smith, eds.) 585–603. Valencia Univ. Press, Spain.