Institute of Mathematical Statistics Collections

Risk and resampling under model uncertainty

Snigdhansu Chatterjee and Nitai D. Mukhopadhyay

Full-text: Open access

Abstract

In statistical exercises where there are several candidate models, the traditional approach is to select one model using some data driven criterion and use that model for estimation, testing and other purposes, ignoring the variability of the model selection process. We discuss some problems associated with this approach. An alternative scheme is to use a model-averaged estimator, that is, a weighted average of estimators obtained under different models, as an estimator of a parameter. We show that the risk associated with a Bayesian model-averaged estimator is bounded as a function of the sample size, when parameter values are fixed. We establish conditions which ensure that a model-averaged estimator’s distribution can be consistently approximated using the bootstrap. A new, data-adaptive, model averaging scheme is proposed that balances efficiency of estimation without compromising applicability of the bootstrap. This paper illustrates that certain desirable risk and resampling properties of model-averaged estimators are obtainable when parameters are fixed but unknown; this complements several studies on minimaxity and other properties of post-model-selected and model-averaged estimators, where parameters are allowed to vary.

Chapter information

Source
Bertrand Clarke and Subhashis Ghosal, eds., Pushing the Limits of Contemporary Statistics: Contributions in Honor of Jayanta K. Ghosh (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2008), 155-169

Dates
First available in Project Euclid: 28 April 2008

Permanent link to this document
https://projecteuclid.org/euclid.imsc/1209398467

Digital Object Identifier
doi:10.1214/074921708000000129

Mathematical Reviews number (MathSciNet)
MR2459223

Subjects
Primary: 60F12
Secondary: 60J05: Discrete-time Markov processes on general state spaces 62C10: Bayesian problems; characterization of Bayes procedures 62F40: Bootstrap, jackknife and other resampling methods

Keywords
bootstrap bounded risk linear regression model averaging model selection

Rights
Copyright © 2008, Institute of Mathematical Statistics

Citation

Chatterjee, Snigdhansu; Mukhopadhyay, Nitai D. Risk and resampling under model uncertainty. Pushing the Limits of Contemporary Statistics: Contributions in Honor of Jayanta K. Ghosh, 155--169, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2008. doi:10.1214/074921708000000129. https://projecteuclid.org/euclid.imsc/1209398467


Export citation

References

  • [1] Andrews, D. W. K. and Guggenberger, P. (2005). Hybrid and size-corrected subsample methods. Cowles Foundation discussion paper # 1605.
  • [2] Andrews, D. W. K. and Guggenberger, P. (2005). The limit of finite sample size and a problem with subsampling. Cowles Foundation discussion paper # 1606.
  • [3] Chatterjee, S. and Bose, A. (2000). Variance estimation in high dimensional regression models. Statist. Sinica 10 497–515.
  • [4] Hall, P. and Wilson, S. R. (1991). Two guidelines for bootstrap hypothesis testing. Biometrics 47 757–762.
  • [5] Hjort, N. L. and Claeskens, G. (2003). Frequentist model average estimators. J. Amer. Statist. Assoc. 98 879–899.
  • [6] Leeb, H. (2006). The distribution of a linear predictor after model selection: unconditional finite sample distributions and asymptotic approximations. IMS Lecture Notes Monograph Series 49 291–311.
  • [7] Leeb, H. and Pötscher, B. M. (2003). The finite sample distribution of post-model-selection estimators and uniform versus non-uniform approximations. Econometric Theory 19 100–142.
  • [8] Leeb, H. and Pötscher, B. M. (2005). Model selection and inference: facts and fiction. Econometric Theory 21 21–59.
  • [9] Leeb, H. and Pötscher, B. M. (2006). Performance limits for the estimators of the risk or distribution of shrinkage type estimators, and some general lower risk bound results. Econometric Theory 22 69–97.
  • [10] Leeb, H. and Pötscher, B. M. (2006). Can one estimate the conditional distribution of post-model-selection estimators? Ann. Statist. 34 2554–2591.
  • [11] Leung, G. and Barron, A. R. (2006). Information theory and mixing least squares regressions. IEEE Trans. Inform. Theory 52 3396–3410.
  • [12] Mammen, E. (1992a). Bootstrap, wild bootstrap and asymptotic normality. Probab. Theory Related Fields 93 439–455.
  • [13] Mammen, E. (1992b). When Does Bootstrap Work: Asymptotic Results and Simulations. Springer, Berlin.
  • [14] Politis, D. N., Romano, J. P. and Wolf, M. (1999). Subsampling. Springer, New York.
  • [15] Pötscher, B. M. (1991). Effects of model selection on inference. Econometric Theory 7 163–185.
  • [16] Pötscher, B. M. (2006). The distribution of model averaging estimators and an impossibility result regarding its estimation. MPRA paper # 73.
  • [17] Raftery, A. E. and Zheng, Y. (2003). Comment on “Frequentist model average estimators,” by N. L. Hjort and G. Claeskens. J. Amer. Statist. Assoc. 98 931–938.
  • [18] Samworth, R. (2003). A note on methods of restoring consistency to the bootstrap. Biometrika 90 985–990.
  • [19] Sethuraman, J. (2004). Are super-efficient estimators super-powerful? Comm. Statist. Theory and Methods 33 2003–2013.
  • [20] Shen, X. and Dougherty, D. P. (2003). Discussion of “Frequentist model average estimators,” by N. L. Hjort and G. Claeskens. J. Amer. Statist. Assoc. 98 917–919.
  • [21] Yang, Y. (2003). Regression with multiple candidate models: selecting or mixing? Statist. Sinica 13 783–809.
  • [22] Yang, Y. (2004). Aggregating regression procedures to improve performance. Bernoulli 10 25–47.
  • [23] Yang, Y. (2005). Can the strengths of AIC and BIC be shared? A conflict between model identification and regression estimation. Biometrika 92 937–950.
  • [24] Yang, Y. (2007). Prediction/estimation with simple linear model: is it really that simple? Econometric Theory 23 1–36.
  • [25] Yuan, Z. and Yang, Y. (2005). Combining linear regression models: when and how? J. Amer. Statist. Assoc. 100 1202–1214.