The Annals of Statistics

Cross-validation in nonparametric regression with outliers

Denis Heng-Yan Leung

Full-text: Open access


A popular data-driven method for choosing the bandwidth in standard kernel regression is cross-validation. Even when there are outliers in the data, robust kernel regression can be used to estimate the unknown regression curve [Robust and Nonlinear Time Series Analysis. Lecture Notes in Statist. (1984) 26 163–184]. However, under these circumstances standard cross-validation is no longer a satisfactory bandwidth selector because it is unduly influenced by extreme prediction errors caused by the existence of these outliers. A more robust method proposed here is a cross-validation method that discounts the extreme prediction errors. In large samples the robust method chooses consistent bandwidths, and the consistency of the method is practically independent of the form in which extreme prediction errors are discounted. Additionally, evaluation of the method’s finite sample behavior in a simulation demonstrates that the proposed method performs favorably. This method can also be applied to other problems, for example, model selection, that require cross-validation.

Article information

Ann. Statist., Volume 33, Number 5 (2005), 2291-2310.

First available in Project Euclid: 25 November 2005

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G08: Nonparametric regression
Secondary: 62F35: Robustness and adaptive procedures 62F40: Bootstrap, jackknife and other resampling methods

Bandwidth cross-validation kernel nonparametric regression robust smoothing


Leung, Denis Heng-Yan. Cross-validation in nonparametric regression with outliers. Ann. Statist. 33 (2005), no. 5, 2291--2310. doi:10.1214/009053605000000499.

Export citation


  • Akaike, H. (1974). A new look at the statistical model identification. IEEE Trans. Automatic Control 19 716--723.
  • Boente, G., Fraiman, R. and Meloche, J. (1997). Robust plug-in bandwidth estimators in nonparametric regression. J. Statist. Plann. Inference 57 109--142.
  • Cleveland, W. S. (1979). Robust locally weighted regression and smoothing scatterplots. J. Amer. Statist. Assoc. 74 829--836.
  • Cox, D. D. (1983). Asymptotics for $M$-type smoothing splines. Ann. Statist. 11 530--551.
  • Gasser, T. and Müller, H.-G. (1979). Kernel estimation of regression functions. Smoothing Techniques for Curve Estimation. Lecture Notes in Math. 757 23--68. Springer, Berlin.
  • Hall, P., Sheather, S., Jones, M. and Marron, J. (1991). On optimal data-based bandwidth selection in kernel density estimation. Biometrika 78 263--269.
  • Härdle, W. (1984). How to determine the bandwidth of some nonlinear smoothers in practice. Robust and Nonlinear Time Series Analysis. Lecture Notes in Statist. 26 163--184. Springer, New York.
  • Härdle, W. (1987). Algorithm AS 222: Resistant smoothing using the fast Fourier transform. Appl. Statist. 36 104--111.
  • Härdle, W. (1990). Applied Nonparametric Regression. Cambridge Univ. Press.
  • Härdle, W. and Gasser, T. (1984). Robust non-parametric function fitting. J. Roy. Statist. Soc. Ser. B 46 42--51.
  • Härdle, W., Hall, P. and Marron, J. (1988). How far are automatically chosen regression smoothing parameters from their optimum? (with discussion). J. Amer. Statist. Assoc. 83 86--101.
  • Härdle, W., Hall, P. and Marron, J. (1992). Regression smoothing parameters that are not far from their optimum. J. Amer. Statist. Assoc. 87 227--233.
  • Härdle, W. and Luckhaus, S. (1984). Uniform consistency of a class of regression function estimators. Ann. Statist. 12 612--623.
  • Härdle, W. and Marron, J. (1983). Optimal bandwidth selection in nonparametric function estimation. Institute of Statistics Mimeo Series no. 1530. Univ. North Carolina, Chapel Hill.
  • Herrmann, E. (2000). Variance estimation and bandwidth selection for kernel regression. In Smoothing and Regression: Approaches, Computation and Application (M. G. Schimek, ed.) 71--107. Wiley, New York.
  • Huber, P. (1964). Robust estimation of a location parameter. Ann. Math. Statist. 35 73--101.
  • Leung, D., Marriott, F. and Wu, E. (1993). Bandwidth selection in robust smoothing. J. Nonparametr. Statist. 2 333--339.
  • Lietz, A. (2000). Analysis of water-quality trends at two discharge stations---one within Big Cypress National Preserve and one near Biscayne Bay---Southern Florida, 1966--1994. U.S. Geological Survey. Available at
  • Marron, J. and Härdle, W. (1983). Random approximations to an error criterion of nonparametric statistics. Institute of Statistics Mimeo Series no. 1538. Univ. North Carolina, Chapel Hill.
  • Nadaraya, E. (1964). On estimating regression. Theory Probab. Appl. 9 141--142.
  • Park, B. and Marron, J. (1990). Comparison of data-driven bandwidth selectors. J. Amer. Statist. Assoc. 85 66--72.
  • Priestley, M. and Chao, M. (1972). Non-parametric function fitting. J. Roy. Statist. Soc. Ser. B 34 385--392.
  • Rice, J. (1984). Bandwidth choice for nonparametric regression. Ann. Statist. 12 1215--1230.
  • Ronchetti, E., Field, C. and Blanchard, W. (1997). Robust linear model selection by cross-validation. J. Amer. Statist. Assoc. 92 1017--1023.
  • Shibata, R. (1981). An optimal selection of regression variables. Biometrika 68 45--54.
  • Wang, F. and Scott, D. (1994). The $L_1$ method for robust nonparametric regression. J. Amer. Statist. Assoc. 89 65--76.
  • Watson, G. (1964). Smooth regression analysis. Sankhyā Ser. A 26 359--372.
  • Whittle, P. (1960). Bounds for the moments of linear and quadratic forms in independent variables. Theory Probab. Appl. 5 302--305.