The Annals of Statistics

From ɛ-entropy to KL-entropy: Analysis of minimum information complexity density estimation

Tong Zhang

Full-text: Open access

Enhanced PDF (294 KB)

Abstract

We consider an extension of ɛ-entropy to a KL-divergence based complexity measure for randomized density estimation methods. Based on this extension, we develop a general information-theoretical inequality that measures the statistical complexity of some deterministic and randomized density estimators. Consequences of the new inequality will be presented. In particular, we show that this technique can lead to improvements of some classical results concerning the convergence of minimum description length and Bayesian posterior distributions. Moreover, we are able to derive clean finite-sample convergence bounds that are not obtainable using previous approaches.

Article information

Source
Ann. Statist., Volume 34, Number 5 (2006), 2180-2210.

Dates
First available in Project Euclid: 23 January 2007

Permanent link to this document
https://projecteuclid.org/euclid.aos/1169571794

Digital Object Identifier
doi:10.1214/009053606000000704

Mathematical Reviews number (MathSciNet)
MR2291497

Zentralblatt MATH identifier
1106.62005

Subjects
Primary: 62C10: Bayesian problems; characterization of Bayes procedures 62G07: Density estimation

Keywords
Bayesian posterior distribution minimum description length density estimation

Citation

Zhang, Tong. From ɛ -entropy to KL-entropy: Analysis of minimum information complexity density estimation. Ann. Statist. 34 (2006), no. 5, 2180--2210. doi:10.1214/009053606000000704. https://projecteuclid.org/euclid.aos/1169571794


Export citation

References

  • Barron, A. and Cover, T. (1991). Minimum complexity density estimation. IEEE Trans. Inform. Theory 37 1034--1054.
    Mathematical Reviews (MathSciNet): MR1111806
    Digital Object Identifier: doi:10.1109/18.86996
    Zentralblatt MATH: 0743.62003
  • Barron, A., Schervish, M. J. and Wasserman, L. (1999). The consistency of posterior distributions in nonparametric problems. Ann. Statist. 27 536--561.
    Mathematical Reviews (MathSciNet): MR1714718
    Digital Object Identifier: doi:10.1214/aos/1017939142
    Project Euclid: euclid.aos/1018031206
    Zentralblatt MATH: 0980.62039
  • Catoni, O. (2004). A PAC-Bayesian approach to adaptive classification. Available at www.proba.jussieu.fr/users/catoni/homepage/classif.pdf.
  • Ghosal, S., Ghosh, J. K. and van der Vaart, A. W. (2000). Convergence rates of posterior distributions. Ann. Statist. 28 500--531.
    Mathematical Reviews (MathSciNet): MR1790007
    Digital Object Identifier: doi:10.1214/aos/1016218228
    Project Euclid: euclid.aos/1016218228
  • Le Cam, L. (1973). Convergence of estimates under dimensionality restrictions. Ann. Statist. 1 38--53.
    Mathematical Reviews (MathSciNet): MR0334381
    Digital Object Identifier: doi:10.1214/aos/1193342380
    Project Euclid: euclid.aos/1193342380
  • Le Cam, L. (1986). Asymptotic Methods in Statistical Decision Theory. Springer, New York.
    Mathematical Reviews (MathSciNet): MR0856411
    Zentralblatt MATH: 0605.62002
  • Li, J. (1999). Estimation of mixture models. Ph.D. dissertation, Dept. Statistics, Yale Univ.
  • Meir, R. and Zhang, T. (2003). Generalization error bounds for Bayesian mixture algorithms. J. Mach. Learn. Res. 4 839--860.
    Mathematical Reviews (MathSciNet): MR2075999
    Digital Object Identifier: doi:10.1162/1532443041424300
    Zentralblatt MATH: 1083.68096
  • Rényi, A. (1961). On measures of entropy and information. Proc. Fourth Berkeley Symp. Math. Statist. Probab. 1 547--561. Univ. California Press, Berkeley.
    Mathematical Reviews (MathSciNet): MR0132570
  • Rissanen, J. (1989). Stochastic Complexity in Statistical Inquiry. World Scientific, Singapore.
    Mathematical Reviews (MathSciNet): MR1082556
    Zentralblatt MATH: 0800.68508
  • Seeger, M. (2002). PAC-Bayesian generalization error bounds for Gaussian process classification. J. Mach. Learn. Res. 3 233--269.
    Mathematical Reviews (MathSciNet): MR1971338
    Digital Object Identifier: doi:10.1162/153244303765208386
  • Shen, X. and Wasserman, L. (2001). Rates of convergence of posterior distributions. Ann. Statist. 29 687--714.
    Mathematical Reviews (MathSciNet): MR1865337
    Digital Object Identifier: doi:10.1214/aos/1009210686
    Project Euclid: euclid.aos/1009210686
    Zentralblatt MATH: 1041.62022
  • van de Geer, S. (2000). Empirical Processes in $M$-Estimation. Cambridge Univ. Press.
  • van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes. With Applications to Statistics. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1385671
    Zentralblatt MATH: 0862.60002
  • Walker, S. and Hjort, N. (2001). On Bayesian consistency. J. R. Stat. Soc. Ser. B Stat. Methodol. 63 811--821.
    Mathematical Reviews (MathSciNet): MR1872068
    Digital Object Identifier: doi:10.1111/1467-9868.00314
    JSTOR: links.jstor.org
    Zentralblatt MATH: 0987.62021
  • Yang, Y. and Barron, A. (1999). Information-theoretic determination of minimax rates of convergence. Ann. Statist. 27 1564--1599.
    Mathematical Reviews (MathSciNet): MR1742500
    Digital Object Identifier: doi:10.1214/aos/1017939142
    Project Euclid: euclid.aos/1017939142
    Zentralblatt MATH: 0978.62008
  • Zhang, T. (1999). Theoretical analysis of a class of randomized regularization methods. In Proc. Twelfth Annual Conference on Computational Learning Theory 156--163. ACM Press, New York.
    Mathematical Reviews (MathSciNet): MR1811611
    Digital Object Identifier: doi:10.1145/307400.307433
  • Zhang, T. (2004). Learning bounds for a generalized family of Bayesian posterior distributions. In Advances in Neural Information Processing Systems 16 (S. Thrun, L. K. Saul and B. Schölkopf, eds.) 1149--1156. MIT Press, Cambridge, MA.

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more