The Annals of Statistics
- Ann. Statist.
- Volume 27, Number 5 (1999), 1564-1599.
Information-theoretic determination of minimax rates of convergence
Full-text: Open access
Abstract
We present some general results determining minimax bounds on statistical risk for density estimation based on certain information-theoretic considerations. These bounds depend only on metric entropy conditions and are used to identify the minimax rates of convergence.
Article information
Source
Ann. Statist., Volume 27, Number 5 (1999), 1564-1599.
Dates
First available in Project Euclid: 23 September 2004
Permanent link to this document
https://projecteuclid.org/euclid.aos/1017939142
Digital Object Identifier
doi:10.1214/aos/1017939142
Mathematical Reviews number (MathSciNet)
MR2001g:62006
Zentralblatt MATH identifier
0978.62008
Subjects
Primary: 62G07: Density estimation
Secondary: 62B10: Information-theoretic topics [See also 94A17] 62C20: Minimax procedures 94A29: Source coding [See also 68P30]
Keywords
Minimax risk density estimation metric entropy Kullback-Leibler distance
Citation
Yang, Yuhong; Barron, Andrew. Information-theoretic determination of minimax rates of convergence. Ann. Statist. 27 (1999), no. 5, 1564--1599. doi:10.1214/aos/1017939142. https://projecteuclid.org/euclid.aos/1017939142
References
- Ball, K. and Pajor, A. (1990). The entropy of convex bodies with "few" extreme points. In Geometry of Banach Spaces 26-32. Cambridge Univ. Press.
- Barron, A. R. (1987). Are Bayes rules consistent in information? In Open Problems in Communication and Computation (T. M. Cover and B. Gopinath, eds.) 85-91. Springer, New York.Mathematical Reviews (MathSciNet): MR922073
- Barron, A. R. (1991). Neural net approximation. In Proceedings of the Yale Workshop on Adaptive Learning Systems (K. Narendra, ed.) Yale University.
- Barron, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39 930-945.
- Barron, A. R. (1994). Approximation and estimation bounds for artificial neural networks. Machine Learning 14 115-133. Zentralblatt MATH: 0818.68127
- Barron, A. R., Birg´e, L. and Massart, P. (1999). Riskbounds for model selection via penalization. Probab. Theory Related Fields 113 301-413. Mathematical Reviews (MathSciNet): MR2000k:62049
Zentralblatt MATH: 0946.62036
Digital Object Identifier: doi:10.1007/s004400050210 - Barron, A. R. and Cover, T. M. (1991). Minimum complexity density estimation. IEEE Trans. Inform. Theory 37 1034-1054. Zentralblatt MATH: 0743.62003
Mathematical Reviews (MathSciNet): MR1111806
Digital Object Identifier: doi:10.1109/18.86996 - Barron, A. R. and Hengartner, N. (1998). Information theory and superefficiency. Ann. Statist. 26 1800-1825. Mathematical Reviews (MathSciNet): MR2000c:62030
Zentralblatt MATH: 0932.62005
Digital Object Identifier: doi:10.1214/aos/1024691358
Project Euclid: euclid.aos/1024691358 - Barron, A. R. and Sheu, C.-H. (1991). Approximation of density functions by sequences of exponential families. Ann. Statist. 19 1347-1369. Zentralblatt MATH: 0739.62027
Mathematical Reviews (MathSciNet): MR1126328
Digital Object Identifier: doi:10.1214/aos/1176348252
Project Euclid: euclid.aos/1176348252 - Bickel, P. J. and Ritov, Y. (1988). Estimating integrated squared density derivatives: sharp best order of convergence estimates. Sankhy¯a Ser. A 50 381-393. Mathematical Reviews (MathSciNet): MR91e:62079
- Birg´e, L. (1983). Approximation dans les espaces metriques et theorie de l'estimation. Z. Wahrsch. Verw. Gebiete 65 181-237.
- Birg´e, L. (1986). On estimating a density using Hellinger distance and some other strange facts. Probab. Theory Related Fields 71 271-291. Mathematical Reviews (MathSciNet): MR87c:62097
Zentralblatt MATH: 0561.62029
Digital Object Identifier: doi:10.1007/BF00332312 - Birg´e, L. and Massart, P. (1993). Rates of convergence for minimum contrast estimators. Probab. Theory Related Fields 97 113-150. Mathematical Reviews (MathSciNet): MR94m:62095
Zentralblatt MATH: 0805.62037
Digital Object Identifier: doi:10.1007/BF01199316 - Birg´e, L. and Massart, P. (1994). Minimum contrast estimators on sieves. Technical report, Univ. Paris-Sud.
- Birg´e, L. and Massart, P. (1995). Estimation of integral functionals of a density. Ann. Statist. 23 11-29. Mathematical Reviews (MathSciNet): MR96c:62065
Digital Object Identifier: doi:10.1214/aos/1176324452
Project Euclid: euclid.aos/1176324452 - Birg´e, L. and Massart, P. (1996). From model selection to adaptive estimation. In Research Papers in Probability and Statistics: Festschrift in Honor of Lucien Le Gam (D. Pollard, E. Torgersen and G. Yang, eds.) 55-87. Springer, New York.
- Birman, M. S. and Solomjak, M. (1974). Quantitative analysis in Sobolev embedding theorems and application to spectral theory. Tenth Math. School Kiev 5-189. Mathematical Reviews (MathSciNet): MR482138
- Bretagnolle, J. and Huber, C. (1979). Estimation des densites: risque minimax. Z. Wahrsch. Verw. Gebiete 47 119-137.
- Carl, B. (1981). Entropy numbers of embedding maps between Besov spaces with an application to eigenvalue problems. Proc. Roy. Soc. Edinburgh 90A 63-70. Mathematical Reviews (MathSciNet): MR83f:46033
- Cencov, N. N. (1972). Statistical Decision Rules and Optimal Inference. Nauka, Moscow; English Mathematical Reviews (MathSciNet): MR343398
- translation in Amer. Math. Soc. Transl. 53 (1982).
- Clarke, B. and Barron, A. R. (1990). Information-theoretic asymptotics of Bayes methods. IEEE Trans. Inform. Theory 36 453-471. Zentralblatt MATH: 0709.62008
Mathematical Reviews (MathSciNet): MR1053841
Digital Object Identifier: doi:10.1109/18.54897 - Clarke, B. and Barron, A. R. (1994). Jeffrey's prior is asymptotically least favorable under entropy risk. J. Statist. Plann. Inference 41 37-60. Mathematical Reviews (MathSciNet): MR1292146
Zentralblatt MATH: 0820.62006
Digital Object Identifier: doi:10.1016/0378-3758(94)90153-8 - Cover, T. M. and Thomas, J. A. (1991). Elements of Information Theory. Wiley, New York. Mathematical Reviews (MathSciNet): MR92g:94001
- Cox, D. D (1988). Approximation of least squares regression on nested subspaces. Ann. Statist. 16 713-732. Zentralblatt MATH: 0669.62047
Mathematical Reviews (MathSciNet): MR947572
Digital Object Identifier: doi:10.1214/aos/1176350830
Project Euclid: euclid.aos/1176350830 - Davisson, L. (1973). Universal noiseless coding. IEEE Trans. Inform. Theory 19 783-795. Mathematical Reviews (MathSciNet): MR57:5417
Digital Object Identifier: doi:10.1109/TIT.1973.1055092 - Davisson, L. and Leon-Garcia, A. (1980). A source matching approach to finding minimax codes. IEEE Trans. Inform. Theory 26 166-174. Mathematical Reviews (MathSciNet): MR81c:94021
Digital Object Identifier: doi:10.1109/TIT.1980.1056167 - DeVore, R. A. and Lorentz, G. G. (1993). Constructive Approximation. Springer, New York. Mathematical Reviews (MathSciNet): MR95f:41001
- Devroye, L. (1987). A Course in Density Estimation. Birkh¨auser, Boston.
- Donoho, D. L. (1993). Unconditional bases are optimal bases for data compression and for statistical estimation. Appl. Comput. Harmon. Anal. 1 100-115. Mathematical Reviews (MathSciNet): MR94j:94011
Zentralblatt MATH: 0796.62083
Digital Object Identifier: doi:10.1006/acha.1993.1008 - Donoho, D. L. (1996). Unconditional bases and bit-level compression. Technical report 498, Dept. Statistics, Stanford Univ. Mathematical Reviews (MathSciNet): MR98a:94012
Zentralblatt MATH: 0936.62004
Digital Object Identifier: doi:10.1006/acha.1996.0032 - Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1996). Density estimation by wavelet thresholding. Ann. Statist. 24 508-539.Mathematical Reviews (MathSciNet): MR97f:62061
Zentralblatt MATH: 0860.62032
Digital Object Identifier: doi:10.1214/aos/1032894451
Project Euclid: euclid.aos/1032894451 - Donoho, D. L. and Liu, R. C. (1991). Geometrizing rates of convergence II. Ann. Statist. 19 633-667. Mathematical Reviews (MathSciNet): MR93f:62066
Digital Object Identifier: doi:10.1214/aos/1176348114
Project Euclid: euclid.aos/1176348114 - Dudley, R. M. (1987). Universal Donsker classes and metric entropy. Ann. Probab. 15 1306-1326. Mathematical Reviews (MathSciNet): MR88g:60081
Zentralblatt MATH: 0631.60004
Digital Object Identifier: doi:10.1214/aop/1176991978
Project Euclid: euclid.aop/1176991978 - Edmunds, D. E. and Triebel, H. (1987). Entropy numbers and approximation numbers in function spaces. Proc. London Math. Soc. 58 137-152.
- Efroimovich, S. Yu. and Pinsker, M. S. (1982). Estimation of square-integrable probability density of a random variable. Problemy Peredachi Informatsii 18 19-38. Mathematical Reviews (MathSciNet): MR85c:62093
- Fano, R. M. (1961). Transmission of Information: A Statistical Theory of Communication. MIT Press.
- Farrell, R. (1972). On the best obtainable asymptotic rates of convergence in estimation of a density function at a point. Ann. Math. Statist. 43 170-180. Mathematical Reviews (MathSciNet): MR45:9406
Zentralblatt MATH: 0238.62049
Digital Object Identifier: doi:10.1214/aoms/1177692711
Project Euclid: euclid.aoms/1177692711 - Hasminskii, R. Z. (1978). A lower bound on the risks of nonparametric estimates of densities in the uniform metric. Theory Probab. Appl. 23 794-796. Mathematical Reviews (MathSciNet): MR516279
- Hasminskii, R. Z. and Ibragimov, I. A. (1990). On density estimation in the view of Kolmogorov's ideas in approximation theory. Ann. Statist. 18 999-1010. Mathematical Reviews (MathSciNet): MR92b:62049
Digital Object Identifier: doi:10.1214/aos/1176347736
Project Euclid: euclid.aos/1176347736 - Haussler, D. (1997). A general minimax result for relative entropy. IEEE Trans. Inform. Theory 40 1276-1280.
- Haussler, D. and Opper, M. (1997). Mutual information, metric entropy and cumulative relative entropy risk. Ann. Statist. 25 2451-2492. Mathematical Reviews (MathSciNet): MR99b:62006
Zentralblatt MATH: 0920.62007
Digital Object Identifier: doi:10.1214/aos/1030741081
Project Euclid: euclid.aos/1030741081 - Ibragimov, I. A. and Hasminskii, R. Z. (1977). On the estimation of an infinite-dimensional parameter in Gaussian white noise. Soviet Math. Dokl. 18 1307-1309.
- Ibragimov, I. A. and Hasminskii, R. Z. (1978). On the capacity in communication by smooth signals. Soviet Math. Dokl. 19 1043-1047. Zentralblatt MATH: 0432.94009
- Jones, L. K. (1992). A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural networktraining. Ann. Statist. 20 608-613. Mathematical Reviews (MathSciNet): MR1150368
Zentralblatt MATH: 0746.62060
Digital Object Identifier: doi:10.1214/aos/1176348546
Project Euclid: euclid.aos/1176348546 - Kolmogorov, A. N. and Tihomirov, V. M. (1959). -entropy and -capacity of sets in function spaces. Uspekhi Mat. Nauk 14 3-86. Mathematical Reviews (MathSciNet): MR22:2890
- Koo, J. Y. and Kim, W. C. (1996). Wavelet density estimation by approximation of log-densities. Statist. Probab. Lett 26 271-278.
- Le Cam, L. M. (1973). Convergence of estimates under dimensionality restrictions. Ann. Statist. 1, 38-53. Mathematical Reviews (MathSciNet): MR48:12700
Zentralblatt MATH: 0255.62006
Digital Object Identifier: doi:10.1214/aos/1193342380
Project Euclid: euclid.aos/1193342380 - Le Cam, L. M. (1986). Asymptotic Methods in Statistical Decision Theory. Springer, New York.
- Lorentz, G. G. (1966). Metric entropy and approximation. Bull. Amer. Math. Soc. 72 903-937. Mathematical Reviews (MathSciNet): MR34:3173
Zentralblatt MATH: 0158.13603
Digital Object Identifier: doi:10.1090/S0002-9904-1966-11586-0
Project Euclid: euclid.bams/1183528486 - Lorentz, G. G., Golitschek, M. v. and Makovoz, Y. (1996). Constructive Approximation: Advanced Problems. Springer, New York. Mathematical Reviews (MathSciNet): MR97k:41002
- Makovoz, Y. (1996). Random approximants and neural networks. J. Approx. Theory 85 98-109. Mathematical Reviews (MathSciNet): MR97c:41025
Zentralblatt MATH: 0857.41024
Digital Object Identifier: doi:10.1006/jath.1996.0031 - Mitjagin, B. S. (1961). The approximation dimension and bases in nuclear spaces. Uspekhi Mat. Nauk 16 63-132. Mathematical Reviews (MathSciNet): MR152865
- Nemirovskii, A. (1985). Nonparametric estimation of smooth regression functions. J. Comput. System. Sci. 23 1-11. Mathematical Reviews (MathSciNet): MR87k:62070
- Nemirovskii, A., Polyak, B. T. and Tsybakov, A. B. (1985). Rates of convergence of nonparametric estimates of maximum-likelihood type. Probl. Peredachi Inf. 21 17-33. Mathematical Reviews (MathSciNet): MR87j:62071
- Pollard, D. (1993). Hypercubes and minimax rates of convergence. Preprint.
- Rissanen, J. (1984). Universal coding, information, prediction, and estimation. IEEE Trans. Inform. Theory 30 629-636. Mathematical Reviews (MathSciNet): MR85g:94009
Digital Object Identifier: doi:10.1109/TIT.1984.1056936 - Rissanen, J., Speed, T. and Yu, B. (1992). Density estimation by stochastic complexity. IEEE Trans. Inform. Theory 38 315-323. Zentralblatt MATH: 0743.62004
- Smoljak, S. A. (1960). The -entropy of some classes E k s B and W s B in the L2 metric. Dokl. Akad. Nauk SSSR 131 30-33. Mathematical Reviews (MathSciNet): MR23:A461a
- Stone, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Ann. Statist. 10 1040-1053. Zentralblatt MATH: 0511.62048
Mathematical Reviews (MathSciNet): MR673642
Digital Object Identifier: doi:10.1214/aos/1176345969
Project Euclid: euclid.aos/1176345969 - Temlyakov, V. N. (1989). Estimation of the asymptotic characteristics of classes of functions with bounded mixed derivative or difference. Trudy Mat. Inst. Steklov 189 162-197.Mathematical Reviews (MathSciNet): MR999814
- Triebel, H. (1975). Interpolation properties of -entropy and diameters. Geometric characteristics of embedding for function spaces of Sobolev-Besov type. Mat. Sb. 98 27-41. Mathematical Reviews (MathSciNet): MR412801
- Van de Geer, S. (1990). Hellinger consistency of certain nonparametric maximum likelihood estimates. Ann. Statist. 21 14-44. Mathematical Reviews (MathSciNet): MR1212164
Zentralblatt MATH: 0779.62033
Digital Object Identifier: doi:10.1214/aos/1176349013
Project Euclid: euclid.aos/1176349013 - Wong, W. H. and Shen, X. (1995). Probability inequalities for likelihood ratios and convergence rates of sieve MLEs. Ann. Statist. 23 339-362. Zentralblatt MATH: 0829.62002
Mathematical Reviews (MathSciNet): MR1332570
Digital Object Identifier: doi:10.1214/aos/1176324524
Project Euclid: euclid.aos/1176324524 - Yang, Y. (1999). Model selection for nonparametric regression. Statist. Sinica 9 475-500.
- Yang, Y. (1999). Minimax nonparametric classification I: rates of convergence. IEEE Trans. Inform. Theory 45 2271-2284.
- Yang, Y. and Barron, A. R. (1997). Information-theoretic determination of minimax rates of convergence. Technical Report 28, Dept. Statistics, Iowa State Univ.
- Yang, Y. and Barron, A. R. (1998). An asymptotic property of model selection criteria. IEEE Trans. Inform. Theory 44 95-116. Zentralblatt MATH: 0949.62041
Mathematical Reviews (MathSciNet): MR1486651
Digital Object Identifier: doi:10.1109/18.650993 - Yatracos, Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov's entropy. Ann. Statist. 13 768-774. Mathematical Reviews (MathSciNet): MR86m:62080
Zentralblatt MATH: 0576.62057
Digital Object Identifier: doi:10.1214/aos/1176349553
Project Euclid: euclid.aos/1176349553 - Yatracos, Y. G. (1988). A lower bound on the error in nonparametric regression type problems. Ann. Statist. 16 1180-1187. Mathematical Reviews (MathSciNet): MR89h:62075
Zentralblatt MATH: 0651.62028
Digital Object Identifier: doi:10.1214/aos/1176350954
Project Euclid: euclid.aos/1176350954 - Yu, B. (1996). Assouad, Fano, and Le Cam. In Research Papers in Probability and Statistics: Festschrift in Honor of Lucien Le Cam (D. Pollard, E. Torgersen and G. Yang, eds.) 423-435. Springer, New York.
- You have access to this content.
- You have partial access to this content.
- You do not have access to this content.
More like this
- From ɛ-entropy to KL-entropy: Analysis of minimum information complexity density estimation
Zhang, Tong, The Annals of Statistics, 2006 - Minimax Solution of Statistical Decision Problems by Iteration
Nelson, Wayne, The Annals of Mathematical Statistics, 1966 - Density Estimation in the $L^\infty$ Norm for Dependent Data with Applications to the Gibbs Sampler
Yu, Bin, The Annals of Statistics, 1993
- From ɛ-entropy to KL-entropy: Analysis of minimum information complexity density estimation
Zhang, Tong, The Annals of Statistics, 2006 - Minimax Solution of Statistical Decision Problems by Iteration
Nelson, Wayne, The Annals of Mathematical Statistics, 1966 - Density Estimation in the $L^\infty$ Norm for Dependent Data with Applications to the Gibbs Sampler
Yu, Bin, The Annals of Statistics, 1993 - Mixing strategies for density estimation
Yang, Yuhong, The Annals of Statistics, 2000 - Adaptation in log-concave density estimation
Kim, Arlene K. H., Guntuboyina, Adityanand, and Samworth, Richard J., The Annals of Statistics, 2018 - Nonparametric Estimation in the Cox Model
O'Sullivan, Finbarr, The Annals of Statistics, 1993 - Minimax Estimation for Linear Regressions
Radner, R., The Annals of Mathematical Statistics, 1958 - Nonasymptotic bounds for vector quantization in Hilbert spaces
Levrard, Clément, The Annals of Statistics, 2015 - Convex regularization for high-dimensional multiresponse tensor regression
Raskutti, Garvesh, Yuan, Ming, and Chen, Han, The Annals of Statistics, 2019 - Efficient multivariate entropy estimation via $k$-nearest neighbour distances
Berrett, Thomas B., Samworth, Richard J., and Yuan, Ming, The Annals of Statistics, 2019