Annals of Statistics

Consistency of data-driven histogram methods for density estimation and classification

Gábor Lugosi and Andrew Nobel

Full-text: Open access

Abstract

We present general sufficient conditions for the almost sure $L_1$-consistency of histogram density estimates based on data-dependent partitions. Analogous conditions guarantee the almost-sure risk consistency of histogram classification schemes based on data-dependent partitions. Multivariate data are considered throughout.

In each case, the desired consistency requires shrinking cells, subexponential growth of a combinatorial complexity measure and sublinear growth of the number of cells. It is not required that the cells of every partition be rectangles with sides parallel to the coordinate axis or that each cell contain a minimum number of points. No assumptions are made concerning the common distribution of the training vectors.

We apply the results to establish the consistency of several known partitioning estimates, including the $k_n$-spacing density estimate, classifiers based on statistically equivalent blocks and classifiers based on multivariate clustering schemes.

Article information

Source
Ann. Statist., Volume 24, Number 2 (1996), 687-706.

Dates
First available in Project Euclid: 24 September 2002

Permanent link to this document
https://projecteuclid.org/euclid.aos/1032894460

Digital Object Identifier
doi:10.1214/aos/1032894460

Mathematical Reviews number (MathSciNet)
MR1394983

Zentralblatt MATH identifier
0859.62040

Subjects
Primary: 62G07: Density estimation
Secondary: 62H30: Classification and discrimination; cluster analysis [See also 68T10, 91C20]

Keywords
Partitioning rules histogram density estimation histogram classification statistically equivalent blocks Vapnik-Chervonenkis inequality

Citation

Lugosi, Gábor; Nobel, Andrew. Consistency of data-driven histogram methods for density estimation and classification. Ann. Statist. 24 (1996), no. 2, 687--706. doi:10.1214/aos/1032894460. https://projecteuclid.org/euclid.aos/1032894460


Export citation

References

  • ANDERSON, T. W. 1966. Some nonparametric multivariate procedures based on statistically Z. equivalent blocks. In Multivariate Analy sis P. R. Krishnaiah, ed. 5 27. Academic Press, New York. Z.
  • BREIMAN, L., FRIEDMAN, J. H., OLSHEN, R. A. and STONE, C. J. 1984. Classification and Regression Trees. Wadsworth, Belmont, CA. Z.
  • CHEN, X. R. and ZHAO, L. C. 1987. Almost sure L -norm convergence for data-based histogram 1 density estimates. J. Multivariate Anal. 21 179 188. Z.
  • COVER, T. M. 1965. Geometrical and statistical properties of sy stems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers 14 326 334. Z.
  • CSISZAR, I. and KORNER, J. 1981. Information Theory: Coding Theorems for Discrete Memory´ ¨ less Sy stems. Academic Press, New York.
  • DEVROy E, L. 1988. Automatic pattern recognition: a study of the probability of error. IEEE Transactions on Pattern Analy sis and Machine Intelligence 10 530 543. Z.
  • DEVROy E, L. and Gy ORFI, L. 1983. Distribution-free exponential bound on the L error of ¨ 1 partitioning estimates of a regression function. In Proceedings of the Fourth PannonZ ian Sy mposium on Mathematical Statistics F. Konecny, J. Mogy orodi and W. Wertz, ´. eds. 67 76. Akademiai Kiado, Budapest, Hungary. ´ ´ Z.
  • DEVROy E, L. and Gy ORFI, L. 1985. Nonparametric Density Estimation: The L View. Wiley, New ¨ 1 York. Z.
  • GERSHO, A. and GRAY, R. M. 1992. Vector Quantization and Signal Compression. Kluwer, Boston.Z.
  • GESSAMAN, M. P. 1970. A consistent nonparametric multivariate density estimator based on statistically equivalent blocks. Ann. Math. Statist. 41 1344 1346. Z.
  • GORDON, L. and OLSHEN, R. A. 1978. Asy mptotically efficient solutions to the classification problem. Ann. Statist. 6 515 533. Z.
  • GORDON, L. and OLSHEN, R. A. 1980. Consistent nonparametric regression from recursive partitioning schemes. J. Multivariate Anal. 10 611 627. Z.
  • GORDON, L. and OLSHEN, R. 1984. Almost surely consistent nonparametric regression from recursive partitioning schemes. J. Multivariate Anal. 15 147 163. Z.
  • HARTIGAN, J. A. 1975. Clustering Algorithms. Wiley, New York. Z.
  • LINDER, T., LUGOSI, G. and ZEGER, K. 1994. Rates of convergence in the source coding theorem, empirical quantizer design, and universal lossy source coding. IEEE Trans. Inform. Theory 40 1728 1740. Z.
  • LUGOSI, G. and NOBEL, A. B. 1993. Consistency of data-driven histogram methods for density estimation and classification. Technical Report UIUC-BI-93-01, Beckman Institute, Univ. Illinois, Urbana Champaign. Z.
  • MAHALANOBIS, P. C. 1961. A method of fractile graphical analysis. Sankhy a Ser. A 23 41 64. Z.
  • NOBEL, A. 1995. Recursive partitioning to reduce distortion. Technical Report UIUC-BI-95-01, Beckman Institute, Univ. Illinois, Urbana Champaign. Z.
  • NOBEL, A. 1996. Histogram regression estimation using data-dependent partitions. Ann. Statist. Z. 24 3. Z.
  • PARTHASARATHY, K. R. and BHATTACHARy A, P. K. 1961. Some limit theorems in regression theory. Sankhy a Ser. A 23 91 102. Z.
  • PATRICK, E. A. and FISHER, F. P., II 1967. Introduction to the performance of distribution-free conditional risk learning sy stems. Technical Report TR-EE-67-12, Purdue Univ. Z.
  • POLLARD, D. 1984. Convergence of Stochastic Processes. Springer, New York. Z.
  • STONE, C. J. 1977. Consistent nonparametric regression. Ann. Statist. 5 595 645. Z.
  • STONE, C. J. 1985. An asy mptotically optimal histogram selection rule. In Proc. Berkeley Conf. Z. in Honor of Jerzy Ney man and Jack Kiefer L. Le Cam and R. A. Olshen, eds. 2 513 520. Wadsworth, Belmont, CA. Z.
  • VAN Ry ZIN, J. 1973. A histogram method of density estimation. Communications in Statistics 2 493 506. Z.
  • VAPNIK, V. N. and CHERVONENKIS, A. YA. 1971. On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appl. 16 264 280. Z.
  • ZHAO, L. C., KRISHNAIAH, P. R. and CHEN, X. R. 1990. Almost sure L -norm convergence for r data-based histogram estimates. Theory Probab. Appl. 35 396 403.
  • TECHNICAL UNIVERSITY OF BUDAPEST CHAPEL HILL, NORTH CAROLINA 27599-3260 BUDAPEST HUNGARY