The Annals of Statistics

Empirical Bayes selection of wavelet thresholds

Iain M. Johnstone and Bernard W. Silverman

Full-text: Open access


This paper explores a class of empirical Bayes methods for level-dependent threshold selection in wavelet shrinkage. The prior considered for each wavelet coefficient is a mixture of an atom of probability at zero and a heavy-tailed density. The mixing weight, or sparsity parameter, for each level of the transform is chosen by marginal maximum likelihood. If estimation is carried out using the posterior median, this is a random thresholding procedure; the estimation can also be carried out using other thresholding rules with the same threshold. Details of the calculations needed for implementing the procedure are included. In practice, the estimates are quick to compute and there is software available. Simulations on the standard model functions show excellent performance, and applications to data drawn from various fields of application are used to explore the practical performance of the approach.

By using a general result on the risk of the corresponding marginal maximum likelihood approach for a single sequence, overall bounds on the risk of the method are found subject to membership of the unknown function in one of a wide range of Besov classes, covering also the case of f of bounded variation. The rates obtained are optimal for any value of the parameter p in (0,∞], simultaneously for a wide range of loss functions, each dominating the Lq norm of the σth derivative, with σ≥0 and 0<q≤2.

Attention is paid to the distinction between sampling the unknown function within white noise and sampling at discrete points, and between placing constraints on the function itself and on the discrete wavelet transform of its sequence of values at the observation points. Results for all relevant combinations of these scenarios are obtained. In some cases a key feature of the theory is a particular boundary-corrected wavelet basis, details of which are discussed.

Overall, the approach described seems so far unique in combining the properties of fast computation, good theoretical properties and good performance in simulations and in practice. A key feature appears to be that the estimate of sparsity adapts to three different zones of estimation, first where the signal is not sparse enough for thresholding to be of benefit, second where an appropriately chosen threshold results in substantially improved estimation, and third where the signal is so sparse that the zero estimate gives the optimum accuracy rate.

Article information

Ann. Statist., Volume 33, Number 4 (2005), 1700-1752.

First available in Project Euclid: 5 August 2005

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62C12: Empirical decision procedures; empirical Bayes procedures 62G08: Nonparametric regression
Secondary: 62G20: Asymptotic properties 62H35: Image analysis 65T60: Wavelets

Adaptivity Bayesian inference nonparametric regression smoothing sparsity


Johnstone, Iain M.; Silverman, Bernard W. Empirical Bayes selection of wavelet thresholds. Ann. Statist. 33 (2005), no. 4, 1700--1752. doi:10.1214/009053605000000345.

Export citation


  • Abramovich, F., Amato, U. and Angelini, C. (2004). On optimality of Bayesian wavelet estimators. Scand. J. Statist. 31 217–234.
  • Abramovich, F. and Benjamini, Y. (1995). Thresholding of wavelet coefficients as a multiple hypotheses testing procedure. Wavelets and Statistics. Lecture Notes in Statist. 103 5–14. Springer, Berlin.
  • Abramovich, F., Benjamini, Y., Donoho, D. and Johnstone, I. (2005). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. To appear. Available at
  • Abramovich, F., Sapatinas, T. and Silverman, B. W. (1998). Wavelet thresholding via a Bayesian approach. J. R. Stat. Soc. Ser. B Stat. Methodol. 60 725–749.
  • Abramovich, F. and Silverman, B. W. (1998). Wavelet decomposition approaches to statistical inverse problems. Biometrika 85 115–129.
  • Antoniadis, A., Jansen, M., Johnstone, I. M. and Silverman, B. W. (2004). EbayesThresh: MATLAB software for Empirical Bayes thresholding. Available at
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • Birgé, L. and Massart, P. (2001). Gaussian model selection. J. Eur. Math. Soc. 3 203–268.
  • Cai, T. T. and Silverman, B. W. (2001). Incorporating information on neighboring coefficients into wavelet estimation. Sankhyā Ser. B 63 127–148.
  • Chipman, H. A., Kolaczyk, E. D. and McCulloch, R. E. (1997). Adaptive Bayesian wavelet shrinkage. J. Amer. Statist. Assoc. 92 1413–1421.
  • Clyde, M. and George, E. I. (2000). Flexible empirical Bayes estimation for wavelets. J. R. Stat. Soc. Ser. B Stat. Methodol. 62 681–698.
  • Clyde, M., Parmigiani, G. and Vidakovic, B. (1998). Multiple shrinkage and subset selection in wavelets. Biometrika 85 391–401.
  • Cohen, A., Daubechies, I. and Vial, P. (1993). Wavelets on the interval and fast wavelet transforms. Appl. Comput. Harmon. Anal. 1 54–81.
  • Coifman, R. R. and Donoho, D. L. (1995). Translation-invariant de-noising. Wavelets and Statistics. Lecture Notes in Statist. 103 125–150. Springer, Berlin.
  • Daubechies, I. (1992). Ten Lectures on Wavelets. SIAM, Philadelphia.
  • Delyon, B. and Juditsky, A. (1996). On minimax wavelet estimators. Appl. Comput. Harmon. Anal. 3 215–228.
  • Donoho, D. L. and Johnstone, I. M. (1994). Spatial adaptation via wavelet shrinkage. Biometrika 81 425–455.
  • Donoho, D. L. and Johnstone, I. M. (1995). Adapting to unknown smoothness via wavelet shrinkage. J. Amer. Statist. Assoc. 90 1200–1224.
  • Donoho, D. L. and Johnstone, I. M. (1999). Asymptotic minimaxity of wavelet estimators with sampled data. Statist. Sinica 9 1–32.
  • Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1995). Wavelet shrinkage: Asymptopia? (with discussion). J. Roy. Statist. Soc. Ser. B 57 301–369.
  • Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1997). Universal near minimaxity of wavelet shrinkage. In Festschrift for Lucien Le Cam (D. Pollard, E. Torgersen and G. L. Yang, eds.) 183–218. Springer, Berlin.
  • Efromovich, S. (1999). Quasi-linear wavelet estimation. J. Amer. Statist. Assoc. 94 189–204.
  • George, E. I. and Foster, D. P. (1998). Empirical Bayes variable selection. In Proc. Workshop on Model Selection. Special Issue of Rassegna di Metodi Statistici ed Applicazioni (W. Racugno, ed.) 79–108. Pitagora Editrice, Bologna.
  • George, E. I. and Foster, D. P. (2000). Calibration and empirical Bayes variable selection. Biometrika 87 731–748.
  • Gopinath, R. A. and Burrus, C. S. (1992). On the moments of the scaling function $\psi_0$. In Proc. 1992 IEEE International Symposium on Circuits and Systems 2 963–966. IEEE Press, Piscataway, NJ.
  • Johnstone, I. M. (1999). Wavelet shrinkage for correlated data and inverse problems: Adaptivity results. Statist. Sinica 9 51–83.
  • Johnstone, I. M. (2003). Threshold selection in transform shrinkage. In Statistical Challenges in Modern Astronomy III (E. D. Feigelson and G. J. Babu, eds.) 343–360. Springer, New York.
  • Johnstone, I. M. (2004). Function estimation and Gaussian sequence models. Draft of a monograph.
  • Johnstone, I. M. and Silverman, B. W. (1997). Wavelet threshold estimators for data with correlated noise. J. Roy. Statist. Soc. Ser. B 59 319–351.
  • Johnstone, I. M. and Silverman, B. W. (1998). Empirical Bayes approaches to mixture problems and wavelet regression. Technical report, Dept. Statistics, Stanford Univ.
  • Johnstone, I. M. and Silverman, B. W. (2004). Boundary coiflets for wavelet shrinkage in function estimation. J. Appl. Probab. 41A 81–98.
  • Johnstone, I. M. and Silverman, B. W. (2004). Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences. Ann. Statist. 32 1594–1649.
  • Johnstone, I. M. and Silverman, B. W. (2005). EbayesThresh: R programs for empirical Bayes thresholding. J. Statist. Software 12(8) 1–38. With accompanying software and manual.
  • Liang, K.-Y. and Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika 73 13–22.
  • Mallat, S. (1999). A Wavelet Tour of Signal Processing, 2nd expanded ed. Academic Press, San Diego, CA.
  • Meyer, Y. (1992). Wavelets and Operators. Cambridge Univ. Press.
  • Müller, P. and Vidakovic, B., eds. (1999). Bayesian Inference in Wavelet-Based Models. Lecture Notes in Statist. 141. Springer, New York.
  • Nason, G. P. (1996). Wavelet shrinkage using cross-validation. J. Roy. Statist. Soc. Ser. B 58 463–479.
  • Nason, G. P. (1998). WaveThresh3 Software. Dept. Mathematics, Univ. Bristol, UK. Available from the CRAN Archive.
  • Paul, D. (2004). Adaptive estimation in linear inverse problems using penalized model selection. Technical report, Dept. Statistics, Stanford Univ.
  • Pensky, M. (2005). Frequentist optimality of Bayesian wavelet shrinkage rules for Gaussian and non-Gaussian noise. Ann. Statist. To appear.
  • Polzehl, J. and Spokoiny, V. (2000). Adaptive weights smoothing with applications to image restoration. J. R. Stat. Soc. Ser. B Stat. Methodol. 62 335–354.
  • Portilla, J., Strela, V., Wainwright, M. J. and Simoncelli, E. P. (2003). Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans. Image Process. 12 1338–1351.
  • R Development Core Team (2004). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Available at
  • Silverman, B. W. (1999). Wavelets in statistics: Beyond the standard assumptions. R. Soc. Lond. Philos. Trans. Ser. A Math. Phys. Eng. Sci. 357 2459–2473.
  • Triebel, H. (1983). Theory of Function Spaces. Birkhäuser, Basel.
  • Vidakovic, B. (1998). Wavelet-based nonparametric Bayes methods. Practical Nonparametric and Semiparametric Bayesian Statistics. Lecture Notes in Statist. 133 133–155. Springer, New York.
  • Vidakovic, B. (1999). Statistical Modeling by Wavelets. Wiley, New York.
  • Wainwright, M. J., Simoncelli, E. P. and Willsky, A. S. (2001). Random cascades on wavelet trees and their use in analyzing and modeling natural images. Appl. Comput. Harmon. Anal. 11 89–123.
  • Zhang, C.-H. (2005). General empirical Bayes wavelet methods and exactly adaptive minimax estimation. Ann. Statist. 33 54–100.