Electronic Journal of Statistics

Copula calibration

Johanna F. Ziegel and Tilmann Gneiting

Full-text: Open access


We propose notions of calibration for probabilistic forecasts of general multivariate quantities. Probabilistic copula calibration is a natural analogue of probabilistic calibration in the univariate setting. It can be assessed empirically by checking for the uniformity of the copula probability integral transform (CopPIT), which is invariant under coordinate permutations and coordinatewise strictly monotone transformations of the predictive distribution and the outcome. The CopPIT histogram can be interpreted as a generalization and variant of the multivariate rank histogram, which has been used to check the calibration of ensemble forecasts. Kendall calibration is an analogue of marginal calibration in the univariate case. Methods and tools are illustrated in simulation studies and applied to compare raw numerical model and statistically postprocessed ensemble forecasts of bivariate wind vectors.

Article information

Electron. J. Statist., Volume 8, Number 2 (2014), 2619-2638.

First available in Project Euclid: 11 December 2014

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62

Copula Kendall distribution multivariate calibration density forecast evaluation ensemble prediction


Ziegel, Johanna F.; Gneiting, Tilmann. Copula calibration. Electron. J. Statist. 8 (2014), no. 2, 2619--2638. doi:10.1214/14-EJS964. https://projecteuclid.org/euclid.ejs/1418313582

Export citation


  • Barbe, P., Genest, C., Ghoudi, K. and Rémillard, B. (1996). On Kendall’s process., J. Multivariate Anal. 58 197–229.
  • Czado, C., Gneiting, T. and Held, L. (2009). Predictive model assessment for count data., Biometrics 65 1254–1261.
  • Dawid, A. P. (1984). Statistical theory: The prequential approach., J. R. Stat. Soc. A 147 278–290.
  • Diebold, F. X., Gunther, T. A. and Tay, A. S. (1998). Evaluating density forecasts with applications to financial risk management., Int. Econ. Rev. 39 863–883.
  • Diebold, F. X., Hahn, J. and Tay, A. S. (1999). Multivariate density forecast evaluation and calibration in financial risk management: High-frequency returns on foreign exchange., Rev. Econ. Stat. 81 661–673.
  • Diks, C., Panchenko, V. and van Dijk, D. (2010). Out-of-sample comparisons of copula specifications in multivariate density forecasts., J. Econ. Dyn. Contr. 34 1596–1609.
  • Eckel, F. A. and Mass, C. F. (2005). Aspects of effective mesoscale, short-range ensemble forecasting., Wea. Forecasting 20 328–350.
  • Ehm, W. (2011). Unbiased risk estimation and scoring rules., C. R. Math. 349 699–702.
  • Genest, C., Nešlehová, J. and Ziegel, J. (2011). Inference in multivariate Archimedean copula models., Test 20 223–256.
  • Gneiting, T., Balabdaoui, F. and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness., J. R. Stat. Soc. B 69 243–268.
  • Gneiting, T. and Katzfuss, M. (2014). Probabilistic forecasting., Ann. Rev. Stat. Appl. 1 125–151.
  • Gneiting, T. and Raftery, A. E. (2005). Weather forecasting with ensemble methods., Science 310 248–249.
  • Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation., J. Am. Stat. Assoc. 102 359–378.
  • Gneiting, T. and Ranjan, R. (2013). Combining predictive distributions., El. J. Stat. 7 1747-1782.
  • Gneiting, T., Raftery, A. E., Westveld, A. H. and Goldman, T. (2005). Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation., Mon. Wea. Rev. 133 1098–1118.
  • Gneiting, T., Stanberry, L. I., Grimit, E. P., Held, L. and Johnson, N. A. (2008). Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds., Test 17 211–235.
  • González-Rivera, G. and Yoldas, E. (2012). Autocontour-based evaluation of multivariate predictive densities., Int. J. Forecasting 28 328–342.
  • Gudendorf, G. and Segers, J. (2010). Extreme-value copulas. In, Copula Theory and Its Applications, (P. Jaworski, F. Durante, W. K. Härdle and T. Rychlik, eds.). Lecture Notes in Statistics – Proceedings 198 127–145. Springer Berlin Heidelberg.
  • Huber, P. J. (1985). Projection pursuit., Ann. Stat. 13 435–475.
  • Ishida, I. (2005). Scanning multivariate conditional densities with probability integral transforms. Center for Advanced Research in Finance, University of Tokio, Working Paper, F-045.
  • Leutbecher, M. and Palmer, T. N. (2008). Ensemble forecasting., J. Computat. Phys. 227 3515–3539.
  • McNeil, A. J. and Nešlehová, J. (2009). Multivariate Archimedean copulas, $d$-monotone functions and $l_1$-norm symmetric distributions., Ann. Stat. 37 3059–3097.
  • Nelsen, R. B. (2006)., An Introduction to Copulas, 2nd ed. Springer, New York.
  • Nelsen, R. B., Quesada-Molina, J. J., Rodríguez-Lallena, J. A. and Úbeda Flores, M. (2003). Kendall distribution functions., Stat. Prob. Lett. 65 263–268.
  • Pinson, P. (2013). Wind energy: Forecasting challenges for its operational management., Stat. Sci. 28 564–585.
  • Pinson, P. and Girard, R. (2012). Evaluating the quality of scenarios of short-term wind power generation., Appl. Energy 96 12–20.
  • R Core Team (2013). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria ISBN, 3-900051-07-0.
  • Röpnack, A., Hense, A., Gebhardt, C. and Majewski, D. (2013). Bayesian model verification of NWP ensemble forecasts., Mon. Wea. Rev. 141 375–387.
  • Rosenblatt, M. (1952). Remarks on a multivariate transformation., Ann. Math. Stat. 23 470–472.
  • Rüschendorf, L. (2009). On the distributional transform, Sklar’s theorem, and the empirical copula process., J. Stat. Plann. Infer. 139 3921–3927.
  • Schaake, J., Pailleux, J., Thielen, J., Arritt, R., Hamill, T., Luo, L., Martin, E., McCollor, D. and Pappenberger, F. (2010). Summary of recommendations of the first workshop on Postprocessing and Downscaling Atmospheric Forecasts for Hydrologic Applications held at Météo-France, Toulouse, France, 15–18 June 2009., Atmos. Sci. Let. 11 59–63.
  • Schefzik, R., Thorarinsdottir, T. L. and Gneiting, T. (2013). Uncertainty quantification in complex simulation models using ensemble copula coupling., Stat. Sci. 28 616–640.
  • Schuhen, N., Thorarinsdottir, T. L. and Gneiting, T. (2012). Ensemble model output statistics for wind vectors., Mon. Wea. Rev. 140 3204–3219.
  • Smith, L. A. and Hansen, J. A. (2004). Extending the limits of ensemble forecast verification with the minimum spanning tree histogram., Mon. Wea. Rev. 132 1522–1528.
  • Tawn, J. A. (1988). Bivariate extreme value theory: Models and estimation., Biometrika 75 397–415.
  • Thorarinsdottir, T. L., Scheuerer, M. and Heinz, C. (2014). Assessing the calibration of high-dimensional ensemble forecasts using the rank histogram., J. Comput. Graph. Stat., in press, DOI: 10.1080/10618600.2014.977447.
  • Wilks, D. S. (2004). The minimum spanning tree histogram as a verification tool for multidimensional ensemble forecasts., Mon. Wea. Rev. 132 1329–1340.