The Annals of Statistics

Limits to classification and regression estimation from ergodic processes

Andrew B. Nobel

Full-text: Open access


We answer two open questions concerning the existence of universal schemes for classification and regression estimation from stationary ergodic processes. It is shown that no measurable procedure can produce weakly consistent regression estimates from every bivariate stationary ergodic process, even if the covariate and response variables are restricted to take values in the unit interval. It is further shown that no measurable procedure can produce weakly consistent classification rules from every bivariate stationary ergodic process for which the response variable is binary valued. The results of the paper are derived via reduction arguments and are based in part on recent work concerning density estimaton from ergodic processes.

Article information

Ann. Statist., Volume 27, Number 1 (1999), 262-273.

First available in Project Euclid: 5 April 2002

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G07: Density estimation
Secondary: 60G10: Stationary processes 62M99: None of the above, but in this section

Classification regression ergodic processes counterexamples reduction arguments


Nobel, Andrew B. Limits to classification and regression estimation from ergodic processes. Ann. Statist. 27 (1999), no. 1, 262--273. doi:10.1214/aos/1018031110.

Export citation


  • Adams, T. M. (1997). Families of ergodic processes without consistent density or regression estimates. Preprint.
  • Adams, T. M. and Nobel, A. B. (1998). On density estimation from ergodic processes. Ann. Probab. 26 794-804.
  • Algoet, P. (1992). Universal schemes for prediction, gambling and portfolio selection. Ann.
  • Probab. 20 901-941. Correction: [Ann. Probab. 23 474-478 (1995).]
  • Ash, R. (1972). Real Analysis and Probability. Academic Press, New York.
  • Bailey, D. H. (1976). Sequential schemes for classifying and predicting ergodic processes. Ph.D. dissertation, Dept. Mathematics, Stanford Univ.
  • Barron, A. R. (1985). Logically smooth density estimation. Technical Report TR 56, Dept. Statistics, Stanford Univ.
  • Cheng, B. C. and Robinson, P. M. (1991). Density estimation in strongly dependent non-linear time series. Statist. Sinica 1 335-359.
  • Delecroix, M. (1987). Sur l'estimation et la pr´evision non-param´etrique des processus ergodiques. Ph.D. thesis, Univ. Lille Flandres Artois, Lille, France.
  • Delecroix, M. and Rosa, A. C. (1996). Nonparametric estimation of a regression function and its derivatives under an ergodic hypothesis. J. Nonparametric Statist. 6 367-382.
  • Devroye, L. and Wagner, T. (1980). Distribution-free consistency results in nonparametric discrimination and regression function estimation. Ann. Statist. 8 231-239.
  • Devroye, L., Gy ¨orfi, L. and Lugosi, G. (1996). A Probabalistic Theory of Pattern Recognition. Springer, New York.
  • Gy ¨orfi, L. (1981). Strongly consistent density estimate from ergodic sample. J. Multivariate Anal. 11 81-84.
  • Gy ¨orfi, L., H¨ardle, W., Sarda, P. and Vieu. P. (1989). Nonparametric Curve Estimation from Time Series. Springer, Berlin.
  • Gy ¨orfi, L. and Lugosi, G. (1992). Kernel density estimation from ergodic sample is not universally consistent. Comput. Statist. Data Anal. 14 437-442.
  • Gy ¨orfi, L., Morvai, G. and Yakowitz, S. (1998). Limits to consistent on-line forecasting for ergodic time series. IEEE Trans. Inform. Theory 44 886-892.
  • Hidalgo, J. (1997). Non-parametric estimation with strongly dependent time multivariate time series. J. Time Ser. Anal. 18 95-122.
  • Ho, H. C. (1995). On the strong uniform consistency of density estimation for strongly dependent sequences. Statist. Probab. Lett. 22 149-156.
  • Masry, E. (1996). Multivariate local polynomial regression for time series: uniform strong consistency and rates. J. Time Ser. Anal. 17 571-599.
  • Morvai, G., Yakowitz, S. and Algoet, P. (1997). Weakly convergent nonparametric forecasting of stationary time series. IEEE Trans. Inform. Theory 43 483-498.
  • Morvai, G., Yakowitz, S. and Gy ¨orfi, L. (1996). Nonparametric inference for ergodic, stationary time series. Ann. Statist. 24 370-379.
  • Morvai, G., Kulkarni, S. and Nobel, A. B. (1997). Regression estimation from an individual stationary sequence. Unpublished manuscript.
  • Nobel, A. B., Morvai, G. and Kulkarni, S. (1997). Density estimation from an individual numerical sequence. IEEE Trans. Inform. Theory 44 537-541.
  • Ornstein, D. S. (1978). Guessing the next output of a stationary process. Israel J. Math. 30 292-296.
  • Rosenblatt, M. (1970). Density estimates and Markov sequences. In Nonparametric Techniques in Statistical Inference (M. Puri, ed.) 199-210. Cambridge Univ. Press, London.
  • Rosenblatt, M. (1991). Stochastic Curve Estimation. IMS, Hayward, CA.
  • Roussas, G. (1967). Nonparametric estimation in Markov processes. Ann. Inst. Statist. Math. 21 73-87.
  • Roussas, G. (1969). Nonparametric estimation of the transition distribution function of a Markov process. Ann. Math. Statist. 40 1386-1400.
  • Ryabko, B. Ya. (1988). Prediction of random sequences and universal coding. Problems Inform. Transmission 24 87-96.
  • Spiegelman, C. and Sacks, J. (1980). Consistent window estimation in nonparametric regression. Ann. Statist. 8 240-246.
  • Stone, C. (1977). Consistent nonparametric regression. Ann. Statist. 5 595-620.
  • Yakowitz, S. (1993). Nearest neighbor regression estimation for null-recurrent Markov time series. Stochastic Process. Appl. 48 311-318.
  • Yakowitz, S., Gy ¨orfi, L., Kieffer, J. and Morvai, G. (1997). Strongly consistent nonparametric estimation of smooth regression functions for stationary ergodic sequences. J. Multivariate Anal. To appear.
  • Yakowitz, S. and Heyde, C. (1997). Long range dependency effects with implications for forecasting and queueing inference. Unpublished manuscript.