Statistical Science

Recent Developments in Bootstrap Methodology

A. C. Davison, D. V. Hinkley, and G. A. Young

Full-text: Open access


Ever since its introduction, the bootstrap has provided both a powerful set of solutions for practical statisticians, and a rich source of theoretical and methodological problems for statistics. In this article, some recent developments in bootstrap methodology are reviewed and discussed. After a brief introduction to the bootstrap, we consider the following topics at varying levels of detail: the use of bootstrapping for highly accurate parametric inference; theoretical properties of nonparametric bootstrapping with unequal probabilities; subsampling and the m out of n bootstrap; bootstrap failures and remedies for superefficient estimators; recent topics in significance testing; bootstrap improvements of unstable classifiers and resampling for dependent data. The treatment is telegraphic rather than exhaustive.

Article information

Statist. Sci., Volume 18, Issue 2 (2003), 141-157.

First available in Project Euclid: 19 September 2003

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Bagging bootstrap conditional inference empirical strength probability parametric bootstrap subsampling superefficient estimator tilted distribution time series weighted bootstrap


Davison, A. C.; Hinkley, D. V.; Young, G. A. Recent Developments in Bootstrap Methodology. Statist. Sci. 18 (2003), no. 2, 141--157. doi:10.1214/ss/1063994969.

Export citation


  • Baggerly, K. A. (1998). Empirical likelihood as a goodness-of-fit measure. Biometrika 85 535--547.
  • Barndorff-Nielsen, O. E. (1980). Conditionality resolutions. Biometrika 67 293--310.
  • Barndorff-Nielsen, O. E. (1983). On a formula for the distribution of the maximum likelihood estimator. Biometrika 70 343--365.
  • Barndorff-Nielsen, O. E. (1986). Inference on full or partial parameters based on the standardized signed log likelihood ratio. Biometrika 73 307--322.
  • Barndorff-Nielsen, O. E. and Cox, D. R. (1979). Edgeworth and saddle-point approximations with statistical applications (with discussion). J. Roy. Statist. Soc. Ser. B 41 279--312.
  • Barndorff-Nielsen, O. E. and Cox, D. R. (1994). Inference and Asymptotics. Chapman and Hall, London.
  • Beran R. J. (1986). Simulated power functions. Ann. Statist. 14 151--173.
  • Beran, R. J. (1987). Prepivoting to reduce level error of confidence sets. Biometrika 74 457--468.
  • Beran, R. J. (1988). Prepivoting test statistics: A bootstrap view of asymptotic refinements. J. Amer. Statist. Assoc. 83 687--697.
  • Beran, R. J. (1995). Stein confidence sets and the bootstrap. Statist. Sinica 5 109--127.
  • Beran, R. J. (1997). Diagnosing bootstrap success. Ann. Inst. Statist. Math. 49 l--24.
  • Beran, R. J. (2003). The impact of the bootstrap on statistical algorithms and theory. Statist. Sci. 18 175--184 (this issue).
  • Bickel, P. J. and Freedman, D. A. (1981). Some asymptotic theory for the bootstrap. Ann. Statist. 9 1196--1217.
  • Bickel, P. J. and Ghosh, J. K. (1990). A decomposition for the likelihood ratio statistic and the Bartlett correction--- a Bayesian argument. Ann. Statist. 18 1070--1090.
  • Bickel, P. J., Götze, F. and van Zwet, W. R. (1997). Resampling fewer than $n$ observations: Gains, losses, and remedies for losses. Statist. Sinica 7 1--32.
  • Booth, J. G. and Hobert, J. P. (1998). Standard errors of prediction in generalized linear mixed models. J. Amer. Statist. Assoc. 93 262--272.
  • Brazzale, A. R. (2000). Practical small-sample parametric inference. Ph.D. dissertation, Dept. Mathematics, Swiss Federal Institute of Technology, Lausanne.
  • Breiman, L. (1996a). Heuristics of instability and stabilization in model selection. Ann. Statist. 24 2350--2383.
  • Breiman, L. (1996b). Bagging predictors. Machine Learning 24 123--140.
  • Bretagnolle, J. (1983). Lois limites du bootstrap de certaines fonctionelles. Ann. Inst. H. Poincaré Probab. Statist. 19 281--296.
  • Brumback, B. A. and Rice, J. A. (1998). Smoothing spline models for the analysis of nested and crossed samples of curves (with discussion). J. Amer. Statist. Assoc. 93 961--994.
  • Bühlmann, P. (2002a). Bootstraps for time series. Statist. Sci. 17 52--72.
  • Bühlmann, P. (2002b). Sieve bootstrap with variable-length Markov chains for stationary categorical time series (with discussion). J. Amer. Statist. Assoc. 97 443--471.
  • Bühlmann, P. and Yu, B. (2002). Analyzing bagging. Ann. Statist. 30 927--961.
  • Canty, A. J., Davison, A. C., Hinkley, D. V. and Ventura, V. (2002). Bootstrap diagnostics. Preprint, Institute of Mathematics, Swiss Federal Institute of Technology, Lausanne.
  • Carpenter, J. (1999). Test inversion bootstrap confidence intervals. J. R. Stat. Soc. Ser. B Stat. Methodol. 61 159--172.
  • Corcoran, S. A. (1998). Bartlett adjustment of empirical discrepancy statistics. Biometrika 85 967--972.
  • Cox, D. R. (1980). Local ancillarity. Biometrika 67 279--286.
  • Davison, A. C. (2003). Statistical Models. Cambridge Univ. Press.
  • Davison, A. C. and Hinkley, D. V. (1997). Bootstrap Methods and Their Application. Cambridge Univ. Press.
  • Delgado, M. A. and González Manteiga, W. (2001). Significance testing in nonparametric regression based on the bootstrap. Ann. Statist. 29 1469--1507.
  • DiCiccio, T. J. and Efron, B. (1992). More accurate confidence intervals in exponential families. Biometrika 79 231--245.
  • DiCiccio, T. J. and Efron, B. (1996). Bootstrap confidence intervals (with discussion). Statist. Sci. 11 189--228.
  • DiCiccio, T. J., Martin, M. A. and Stern, S. E. (2001). Simple and accurate one-sided inference from signed roots of likelihood ratios. Canad. J. Statist. 29 67--76.
  • DiCiccio, T. J. and Romano, J. P. (1990). Nonparametric confidence limits by resampling methods and least favorable families. Internat. Statist. Rev. 58 59--76.
  • DiCiccio, T. J. and Romano, J. P. (1995). On bootstrap procedures for second-order accurate confidence limits in parametric models. Statist. Sinica 5 141--160.
  • Durbin, J. (1980). Approximations for densities of sufficient estimators. Biometrika 67 311--333.
  • Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Ann. Statist. 7 1--26.
  • Efron, B. (1983). Estimating the error rate of a prediction rule: Improvement on cross-validation. J. Amer. Statist. Assoc. 78 316--331.
  • Efron, B. (1986). How biased is the apparent error rate of a prediction rule? J. Amer. Statist. Assoc. 81 461--470.
  • Efron, B. (1987). Better bootstrap confidence intervals (with discussion). J. Amer. Statist. Assoc. 82 171--200.
  • Efron, B. and Hinkley, D. V. (1978). Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information (with discussion). Biometrika 65 457--487.
  • Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman and Hall, New York.
  • Efron, B. and Tibshirani, R. J. (1997). Improvements on cross-validation: The $.632+$ bootstrap method. J. Amer. Statist. Assoc. 92 548--560.
  • Efron, B. and Tibshirani, R. J. (1998). The problem of regions. Ann. Statist. 26 1687--1718.
  • Fan, J. and Lin, S. (1998). Test of significance when data are curves. J. Amer. Statist. Assoc. 93 1007--1021.
  • Freund, Y. and Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System Sci. 55 119--139.
  • Garthwaite, P. H. and Buckland, S. T. (1992). Generating Monte Carlo confidence intervals by the Robbins--Monro process. Appl. Statist. 41 159--171.
  • Hall, P. (1985). Resampling a coverage pattern. Stochastic Process. Appl. 20 231--246.
  • Hall, P. (1986). On the bootstrap and confidence intervals. Ann. Statist. 14 1431--1452.
  • Hall, P. (1992). The Bootstrap and Edgeworth Expansion. Springer, New York.
  • Hall, P. and Presnell, B. (1999a). Intentionally biased bootstrap methods. J. R. Stat. Soc. Ser. B Stat. Methodol. 61 143--158.
  • Hall, P. and Presnell, B. (1999b). Biased bootstrap methods for reducing the effects of contamination. J. R. Stat. Soc. Ser. B Stat. Methodol. 61 661--680.
  • Hall, P. and Presnell, B. (1999c). Density estimation under constraints. J. Comput. Graph. Statist. 8 259--277.
  • Hall, P. and Wilson, S. R. (1991). Two guidelines for bootstrap hypothesis testing. Biometrics 47 757--762.
  • Härdle, W. (1989). Resampling for inference from curves. Bull. Inst. Internat. Statist. 53 53--64.
  • Härdle, W. (1990). Applied Nonparametric Regression. Cambridge Univ. Press.
  • Hesterberg, T. C. (1999). Bootstrap tilting confidence intervals and hypothesis tests. In Computer Science and Statistics: Proc. 31st Symposium on the Interface 389--393. Interface Foundation of North America, Inc., Fairfax Station, VA.
  • Hinkley, D. V. (1980). Likelihood as approximate pivotal distribution. Biometrika 67 287--292.
  • Huang, H. (2002). Scenario generation for multivariate series data using the nearest neighbor bootstrap. Ph.D. dissertation, Dept. Decision Sciences and Engineering, Rensselaer Polytechnic Institute, Troy, New York.
  • Lahiri, S. N. (2003). On the impact of bootstrap in survey sampling and small-area estimation. Statist. Sci. 18 199--210 (this issue).
  • Lawless, J. (1982). Statistical Models and Methods for Lifetime Data. Wiley, New York.
  • Lee, S. M. S. and Young, G. A. (2003). Prepivoting by weighted bootstrap iteration. Biometrika 90 393--410.
  • Lee, Y. D. and Lahiri, S. N. (2002). Least squares variogram fitting by spatial subsampling. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 837--854.
  • Liu, R. Y. and Singh, K. (1997). Notions of limiting $P$ values based on data depth and bootstrap. J. Amer. Statist. Assoc. 92 266--277.
  • Mammen, E. (1993). Bootstrap and wild bootstrap for high-dimensional linear models. Ann. Statist. 21 255--285.
  • Martin, M. A. (1990). On bootstrap iteration for coverage correction in confidence intervals. J. Amer. Statist. Assoc. 85 1105--1118.
  • McCullagh, P. (2000). Resampling and exchangeable arrays. Bernoulli 6 285--301.
  • Newton, M. A. and Geyer, C. J. (1994). Bootstrap recycling: A Monte Carlo alternative to the nested bootstrap. J. Amer. Statist. Assoc. 89 905--912.
  • Owen, A. B. (1988). Empirical likelihood ratio confidence intervals for a single functional. Biometrika 75 237--249.
  • Owen, A. B. (2001). Empirical Likelihood. Chapman and Hall/CRC, Boca Raton, FL.
  • Politis, D. N. (2003). The impact of bootstrap methods on time series analysis. Statist. Sci. 18 219--230 (this issue).
  • Politis, D. N., Paparoditis, E. and Romano, J. P. (1999). Resampling marked point processes. In Multivariate Analysis, Design of Experiments, and Survey Sampling (S. Ghosh, ed.) 163--185. Dekker, New York.
  • Politis, D. N., Romano, J. P. and Wolf, M. (1999). Subsampling. Springer, New York.
  • Putter, H. and van Zwet, W. R. (1996). Resampling: Consistency of substitution estimators. Ann. Statist. 24 2297--2318.
  • Rajagopalan, B. and Lall, U. (1999). A $k$-nearest-neighbor simulator for daily precipitation and other weather variables. Water Resources Res. 35 3089--3101.
  • Samworth, R. J. (2003). A note on methods of restoring consistency to the bootstrap. Biometrika. To appear.
  • Schapire, R. E., Freund, Y., Bartlett, P. and Lee, W. S. (1998). Boosting the margin: A new explanation for the effectiveness of voting methods. Ann. Statist. 26 1651--1686.
  • Severini, T. A. (2000). Likelihood Methods in Statistics. Clarendon, Oxford.
  • Shao, J. (2003). Impact of the bootstrap on sample surveys. Statist. Sci. 18 191--198 (this issue).
  • Shao, J. and Tu, D. (1995). The Jackknife and Bootstrap. Springer, New York.
  • Singh, K. (1981). On the asymptotic accuracy of Efron's bootstrap. Ann. Statist. 91 1187--1195.
  • Stute, W., González Manteiga, W. and Presedo Quindimil, M. (1998). Bootstrap approximations in model checks for regression. J. Amer. Statist. Assoc. 93 141--149.
  • Ventura, V. (2002). Non-parametric bootstrap recycling. Statist. Comput. 12 261--273.
  • Wang, Y. D. and Wahba, G. (1995). Bootstrap confidence intervals for smoothing splines and their comparison to Bayesian confidence intervals. J. Statist. Comput. Simulation 51 263--279.
  • Wu, C.-F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis (with discussion). Ann. Statist. 14 1261--1350.