The Annals of Applied Statistics

Nonparametric Bayesian multiple testing for longitudinal performance stratification

James G. Scott

Full-text: Open access


This paper describes a framework for flexible multiple hypothesis testing of autoregressive time series. The modeling approach is Bayesian, though a blend of frequentist and Bayesian reasoning is used to evaluate procedures. Nonparametric characterizations of both the null and alternative hypotheses will be shown to be the key robustification step necessary to ensure reasonable Type-I error performance. The methodology is applied to part of a large database containing up to 50 years of corporate performance statistics on 24,157 publicly traded American companies, where the primary goal of the analysis is to flag companies whose historical performance is significantly different from that expected due to chance.

Article information

Ann. Appl. Stat., Volume 3, Number 4 (2009), 1655-1674.

First available in Project Euclid: 1 March 2010

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Multiple testing Bayesian model selection nonparametric Bayes financial time series


Scott, James G. Nonparametric Bayesian multiple testing for longitudinal performance stratification. Ann. Appl. Stat. 3 (2009), no. 4, 1655--1674. doi:10.1214/09-AOAS252.

Export citation


  • Antoniak, C. (1974). Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Statist. 2 1152–1174.
  • Bartlett, M. (1957). A comment on D.V. Lindley’s statistical paradox. Biometrika 44 533–534.
  • Basu, S. and Chib, S. (2003). Marginal likelihood and Bayes factors for Dirichlet process mixture models. J. Amer. Statist. Assoc. 98 224–235.
  • Berger, J., Pericchi, L. and Varshavsky, J. (1998). Bayes factors and marginal distributions in invariant situations. Sankhya Ser. A 60 307–321.
  • Berger, J. O. and Guglielmi, A. (2001). Bayesian and conditional frequentist testing of parametric model versus nonparametric alternatives. J. Amer. Statist. Assoc. 96 174–184.
  • Berger, J. O. and Pericchi, L. (1996). The intrinsic Bayes factor for model selection and prediction. J. Amer. Statist. Assoc. 91 109–122.
  • Bigelow, J. and Dunson, D. (2005). Semiparametric classification in hierarchical functional data analysis. Technical report, Duke Univ., Dept. Statistical Science.
  • Bowman, E. H. and Helfat, C. E. (2001). Does corporate strategy matter? Strategic Management Journal 22 1–23.
  • Carvalho, C. M. and Scott, J. G. (2009). Objective Bayesian model selection in Gaussian graphical models. Biometrika 96 497–512.
  • Cui, W. and George, E. I. (2008). Empirical Bayes vs. fully Bayes variable selection. J. Statist. Plann. Inference 138 888–900.
  • Dahl, D. B. and Newton, M. A. (2007). Multiple hypothesis testing by clustering treatment effects. J. Amer. Statist. Assoc. 102 517–526.
  • Denrell, J. (2003). Vicarious learning, undersampling of failure, and the myths of management. Organizational Science 4 227–243.
  • Denrell, J. (2005). Selection bias and the perils of benchmarking. Harvard Business Review April, 2005 114–119.
  • Do, K.-A., Muller, P. and Tang, F. (2005). A Bayesian mixture model for differential gene expression. J. Roy. Statist. Soc. Ser. C 54 627–644.
  • Dunson, D. and Herring, A. (2006). Semiparametric Bayesian latent trajectory models. Technical report, Duke Univ., Dept. Statistical Science.
  • Escobar, M. and West, M. (1995). Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90 577–588.
  • Ferguson, T. (1973). A Bayesian analysis of some nonparametric problems. Ann. Statist. 1 209–230.
  • Frühwirth-Schnatter, S. and Kaufmann, S. (2008). Model-based clustering of multiple time series. J. Bus. Econom. Statist. 26 78–89.
  • Gelfand, A., Kottas, A. and MacEachern, S. (2005). Bayesian nonparametric spatial modeling with Dirichlet process mixing. J. Amer. Statist. Assoc. 100 1021–1035.
  • George, E. I. and Foster, D. P. (2000). Calibration and empirical Bayes variable selection. Biometrika 87 731–747.
  • Gramacy, R. (2005). Bayesian treed Gaussian process models. Ph.D. thesis, Univ. California–Santa Cruz.
  • Harrigan, K. (1985). An application of clustering for strategic group analysis. Strategic Management Journal 6 55–73.
  • Hawawini, G., Subramanian, V. and Verdin, P. (2003). Is performance driven by industry- or firm-specific factors? A new look at the evidence. Strategic Management Journal 24 1–16.
  • Ishwaran, H. and James, L. (2001). Gibbs sampling methods for stick-breaking priors. J. Amer. Statist. Assoc. 96 161–173.
  • Jefferys, W. and Berger, J. (1992). Ockham’s razor and Bayesian analysis. Amer. Sci. 80 64–72.
  • Jeffreys, H. (1961). Theory of Probability, 3rd ed. Oxford Univ. Press.
  • Johnstone, I. M. and Silverman, B. W. (2004). Needles and Straw in Haystacks: Empirical-Bayes estimates of possibly sparse sequences. Ann. Statist. 32 1594–1649.
  • Kleinman, K. and Ibrahim, J. (1998). A semiparametric Bayesian approach to the random effects model. Biometrics 54 921–938.
  • Laud, P. and Ibrahim, J. (1995). Predictive model selection. J. Roy. Statist. Soc. Ser. B 57 247–262.
  • Müller, P., West, M. and MacEachern, S. (1997). Bayesian models for non-linear auto-regressions. J. Time Ser. Anal. 18 593–614.
  • O’Hagan, A. (1995). Fractional Bayes factors for model comparison. J. Roy. Statist. Soc. Ser. B 57 99–138.
  • Rasmussen, C. E. and Williams, C. (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.
  • Ruefli, T. W. and Wiggins, R. R. (2000). Longitudinal performance stratification: An iterative Kolmogorov–Smirnov approach. Management Sci. 46 685–692.
  • Ruefli, T. W. and Wiggins, R. R. (2002). Sustained competitive advantage: Temporal dynamics and the incidence and persistence of superior economic performance. Organization Science 13 81–105.
  • Scott, J. G. (2009). Supplement to “Nonparametric Bayesian multiple testing for longitudinal performance stratification.”
  • Scott, J. G. and Berger, J. O. (2006). An exploration of aspects of Bayesian multiple testing. J. Statist. Plann. Inference 136 2144–2162.
  • Scott, J. G. and Berger, J. O. (2008). Bayes and Empirical-Bayes multiplicity adjustment in the variable-selection problem. Discussion Paper 2008-10, Duke Univ., Dept. Statistical Science.
  • Scott, J. G. and Carvalho, C. M. (2008). Feature-inclusion stochastic search for Gaussian graphical models. J. Comput. Graph. Statist. 17 790–808.
  • Wernerfelt, B. (1984). The resource-based view of the firm. Strategic Management Journal 5 171–180.

Supplemental materials